You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2026/02/06 19:02:42 UTC
[PR] feat(blob): Blob schema definition [hudi]
the-other-tim-brown opened a new pull request, #18108:
URL: https://github.com/apache/hudi/pull/18108
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
Defines the blob storage schema and converters to/from Spark as the starting point for building out Blob support in Hudi.
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
- Adds a new logical type to the HoodieSchema that matches the definition in the proposed [RFC-100 changes here](https://github.com/apache/hudi/pull/18013/changes)
- Updates the spark schema converter to attach extra metadata to describe the type and then use that metadata when converting back to HoodieSchema
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
Lays the foundation for future blob storage work
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
Low, does not impact existing paths outside of spark type conversion which is tested
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862171969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3930680643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
}, {
"hash" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"triggerType" : "PUSH"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127",
"triggerID" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* bfbbf46b7473d4f34965b5a311418ff1c678ff6f UNKNOWN
* cae177ea0924da786dc0ab258cbfd11ea31a51a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2828157594
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -630,6 +648,10 @@ public static HoodieSchema.Variant createVariantShredded(String name, String nam
return new HoodieSchema.Variant(recordSchema);
}
+ public static HoodieSchema.Blob createBlob() {
+ return new HoodieSchema.Blob(Blob.DEFAULT_NAME);
+ }
Review Comment:
This is just used when defining the underlying schema for the field and I don't think most users will care too much about how to name the underlying avro representation. I can add a new helper if that is desired.
I am updating the test case that covers the round trip create, serialize, deserialize path to assert that this assumption remains true that the schema of the same name can be reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown merged PR #18108:
URL: https://github.com/apache/hudi/pull/18108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862121558
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3876394408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805403559
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1979,119 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
+ super.validate(schema);
+ if (schema.getType() != Schema.Type.RECORD) {
+ throw new IllegalArgumentException("Blob logical type can only be applied to RECORD schemas, got: " + schema.getType());
+ }
+ if (!schema.getFields().equals(HoodieSchema.Blob.BLOB_FIELDS)) {
+ throw new IllegalArgumentException("Blob logical type cannot be applied to schema: " + schema);
+ }
+ }
+ }
+
+ /**
+ * Factory for creating BlobLogicalType instances.
+ */
+ private static class BlogLogicalTypeFactory implements LogicalTypes.LogicalTypeFactory {
Review Comment:
typo here, should be `BlobLogicalTypeFactory`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898857712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1d06141c80411593418964d6a86e7c1c91630fda Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020)
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862325008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
}, {
"hash" : "466ff742031912b97bb660a31944a7f9ff359275",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "466ff742031912b97bb660a31944a7f9ff359275",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777)
* 466ff742031912b97bb660a31944a7f9ff359275 UNKNOWN
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3888261419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3886422164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863)
* 074d15b66ad8e8bdbb2f2dee9a947eeb944180ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805071988
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -79,20 +80,24 @@ object HoodieSparkSchemaConverters {
// Complex types
case ArrayType(elementType, containsNull) =>
- val elementSchema = toHoodieType(elementType, containsNull, recordName, nameSpace)
+ val elementSchema = toHoodieType(elementType, containsNull, recordName, nameSpace, metadata)
HoodieSchema.createArray(elementSchema)
case MapType(StringType, valueType, valueContainsNull) =>
- val valueSchema = toHoodieType(valueType, valueContainsNull, recordName, nameSpace)
+ val valueSchema = toHoodieType(valueType, valueContainsNull, recordName, nameSpace, metadata)
HoodieSchema.createMap(valueSchema)
+ case _: StructType if metadata.contains(HoodieSchema.TYPE_METADATA_FIELD) &&
+ metadata.getString(HoodieSchema.TYPE_METADATA_FIELD).equalsIgnoreCase(HoodieSchemaType.BLOB.name()) =>
+ // Handle blob struct type
+ HoodieSchema.createBlob()
Review Comment:
Blob detection during Spark -> Hudi conversion is based solely on `StructField` metadata (`hudi_type=BLOB`) and does not validate that the `StructType` actually matches the expected blob structure (type/data/reference...). This can silently accept incompatible schemas and reconstruct them as `HoodieSchema.createBlob()`. Consider validating the structโs fields/types before returning a BLOB schema (or failing fast with an informative error).
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaType.java:
##########
@@ -156,6 +161,8 @@ public static HoodieSchemaType fromAvro(Schema avroSchema) {
return UUID;
} else if (logicalType instanceof VariantLogicalType) {
return VARIANT;
+ } else if (logicalType == HoodieSchema.BlobLogicalType.blob()) {
Review Comment:
`fromAvro` checks blob logical type using reference equality (`logicalType == HoodieSchema.BlobLogicalType.blob()`). This can fail if an Avro schema with `logicalType:"blob"` was parsed before the custom logical type was registered, since Avro may create a different LogicalType instance. Use a more robust check (e.g., `instanceof BlobLogicalType` or name-based comparison) so blob detection doesnโt depend on initialization/parse order.
```suggestion
} else if (logicalType instanceof HoodieSchema.BlobLogicalType) {
```
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -152,20 +159,28 @@ public static HoodieSchema fromAvroSchema(Schema avroSchema) {
if (avroSchema == null) {
return null;
}
+ HoodieSchema schema;
LogicalType logicalType = avroSchema.getLogicalType();
if (logicalType != null) {
if (logicalType instanceof LogicalTypes.Decimal) {
- return new HoodieSchema.Decimal(avroSchema);
+ schema = new HoodieSchema.Decimal(avroSchema);
} else if (logicalType instanceof LogicalTypes.TimeMillis || logicalType instanceof LogicalTypes.TimeMicros) {
- return new HoodieSchema.Time(avroSchema);
+ schema = new HoodieSchema.Time(avroSchema);
} else if (logicalType instanceof LogicalTypes.TimestampMillis || logicalType instanceof LogicalTypes.TimestampMicros
|| logicalType instanceof LogicalTypes.LocalTimestampMillis || logicalType instanceof LogicalTypes.LocalTimestampMicros) {
- return new HoodieSchema.Timestamp(avroSchema);
+ schema = new HoodieSchema.Timestamp(avroSchema);
} else if (logicalType == VariantLogicalType.variant()) {
- return new HoodieSchema.Variant(avroSchema);
+ schema = new HoodieSchema.Variant(avroSchema);
+ } else if (logicalType == BlobLogicalType.blob()) {
Review Comment:
`fromAvroSchema` uses reference equality (`logicalType == BlobLogicalType.blob()`) to detect the blob logical type. If the Avro schema was parsed before `LogicalTypes.register("blob", ...)` ran, `avroSchema.getLogicalType()` may not be the singleton instance and blob detection will fail. Prefer `instanceof BlobLogicalType` or checking `logicalType.getName()` to make detection independent of registration/parse ordering.
```suggestion
} else if (logicalType instanceof BlobLogicalType) {
```
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -54,7 +54,8 @@ object HoodieSparkSchemaConverters {
def toHoodieType(catalystType: DataType,
nullable: Boolean = false,
recordName: String = "topLevelRecord",
- nameSpace: String = ""): HoodieSchema = {
+ nameSpace: String = "",
+ metadata: Metadata = Metadata.empty): HoodieSchema = {
Review Comment:
`toHoodieType` gained a new `metadata` parameter, which is a binary-incompatible API change for any already-compiled callers (Scala/Java) expecting the previous 4-arg signature. Consider adding an overload with the old signature delegating to the new method (passing `Metadata.empty`) to preserve binary compatibility while still enabling blob metadata handling.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +964,64 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (getType() == HoodieSchemaType.RECORD) {
Review Comment:
`containsBlobType` only descends into `UNION` and `RECORD`, but not `VARIANT` even though `VARIANT` can contain nested schemas (e.g., shredded variant `typed_value`). This means blob types nested under VARIANT wonโt be detected, allowing arrays/maps that indirectly contain blobs to bypass the new validations. Consider traversing any schema that `hasFields()` (RECORD/VARIANT/BLOB) instead of only RECORD, and update `validateNoBlobsInArrayOrMap` accordingly.
```suggestion
} else if (hasFields()) {
```
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##########
@@ -1689,4 +1690,221 @@ public void testVariantLogicalTypeSingleton() {
// Verify the logical type name
assertEquals("variant", instance1.getName());
}
+
+ // ==================== Blob Type Tests ====================
+
+ @Test
+ public void testCreateBlob() {
+ HoodieSchema.Blob blob = HoodieSchema.createBlob();
+
+ assertNotNull(blob);
+ assertEquals("blob", blob.getName());
+ assertEquals(HoodieSchemaType.BLOB, blob.getType());
+
+ // Verify storage_type field
+ Option<HoodieSchemaField> storageTypeOpt = blob.getField("type");
+ assertTrue(storageTypeOpt.isPresent());
+ HoodieSchemaField storageTypeField = storageTypeOpt.get();
+ assertEquals(HoodieSchemaType.ENUM, storageTypeField.schema().getType());
+ assertEquals(Arrays.asList("INLINE", "OUT_OF_LINE"), storageTypeField.schema().getEnumSymbols());
+ assertFalse(storageTypeField.schema().isNullable());
+
+ // Verify data field is nullable
+ Option<HoodieSchemaField> dataOpt = blob.getField("data");
+ assertTrue(dataOpt.isPresent());
+ HoodieSchemaField dataField = dataOpt.get();
+ assertTrue(dataField.schema().isNullable());
+ assertEquals(HoodieSchemaType.BYTES, dataField.schema().getNonNullType().getType());
+
+ // Verify reference field is nullable
+ Option<HoodieSchemaField> refOpt = blob.getField("reference");
+ assertTrue(refOpt.isPresent());
+ HoodieSchemaField refField = refOpt.get();
+ assertTrue(refField.schema().isNullable());
+ assertEquals(HoodieSchemaType.RECORD, refField.schema().getNonNullType().getType());
+
+ HoodieSchema refSchema = refOpt.get().schema().getNonNullType();
+ assertEquals(HoodieSchemaType.RECORD, refSchema.getType());
+
+ // Verify reference has all required fields
+ Option<HoodieSchemaField> externalPathOpt = refSchema.getField("external_path");
+ assertTrue(externalPathOpt.isPresent());
+ assertEquals(HoodieSchemaType.STRING, externalPathOpt.get().schema().getType());
+ assertFalse(externalPathOpt.get().schema().isNullable());
+
+ Option<HoodieSchemaField> offsetOpt = refSchema.getField("offset");
+ assertTrue(offsetOpt.isPresent());
+ assertEquals(HoodieSchemaType.LONG, offsetOpt.get().schema().getNonNullType().getType());
+ assertTrue(offsetOpt.get().schema().isNullable());
+
+ Option<HoodieSchemaField> lengthOpt = refSchema.getField("length");
+ assertTrue(lengthOpt.isPresent());
+ assertEquals(HoodieSchemaType.LONG, lengthOpt.get().schema().getNonNullType().getType());
+ assertTrue(lengthOpt.get().schema().isNullable());
+
+ Option<HoodieSchemaField> managedOpt = refSchema.getField("managed");
+ assertTrue(managedOpt.isPresent());
+ assertEquals(HoodieSchemaType.BOOLEAN, managedOpt.get().schema().getType());
+ assertFalse(managedOpt.get().schema().isNullable());
+ }
+
+ @Test
+ public void testBlobLogicalTypeDetection() {
+ HoodieSchema.Blob blob = HoodieSchema.createBlob();
+
+ // Verify logical type is set
+ Schema avroSchema = blob.toAvroSchema();
+ assertNotNull(avroSchema.getLogicalType());
+ assertEquals("blob", avroSchema.getLogicalType().getName());
+
+ // Verify it can be detected back
+ HoodieSchema reconstructed = HoodieSchema.fromAvroSchema(avroSchema);
+ assertInstanceOf(HoodieSchema.Blob.class, reconstructed);
+ assertEquals(HoodieSchemaType.BLOB, reconstructed.getType());
+ }
+
+ @Test
+ public void testBlobAsRecordField() {
+ HoodieSchema recordSchema = HoodieSchema.createRecord("test_record", null, null, Arrays.asList(
+ HoodieSchemaField.of("id", HoodieSchema.create(HoodieSchemaType.INT)),
+ HoodieSchemaField.of("file_data", HoodieSchema.createBlob())
+ ));
+
+ Option<HoodieSchemaField> blobFieldOpt = recordSchema.getField("file_data");
+ assertTrue(blobFieldOpt.isPresent());
+ HoodieSchemaField blobField = blobFieldOpt.get();
+ assertEquals(HoodieSchemaType.BLOB, blobField.schema().getType());
+
+ // Verify the blob field has proper structure
+ HoodieSchema blobSchema = blobField.schema();
+ assertInstanceOf(HoodieSchema.Blob.class, blobSchema);
+
+ // Validate the blob schema can be converted to string and back without losing the logical type
+ String recordJson = recordSchema.toString();
+ HoodieSchema parsedRecord = HoodieSchema.parse(recordJson);
+ Option<HoodieSchemaField> parsedBlobFieldOpt = parsedRecord.getField("file_data");
+ assertTrue(parsedBlobFieldOpt.isPresent());
+ HoodieSchemaField parsedBlobField = parsedBlobFieldOpt.get();
+ assertInstanceOf(HoodieSchema.Blob.class, parsedBlobField.schema());
+ assertEquals(HoodieSchemaType.BLOB, parsedBlobField.schema().getType());
+ }
+
+ private static final String BLOB_JSON = "{"
+ + "\"type\":\"record\","
+ + "\"name\":\"blob\","
+ + "\"logicalType\":\"blob\","
+ + "\"fields\":["
+ + " {\"name\":\"storage_type\",\"type\":\"string\"},"
+ + " {\"name\":\"data\",\"type\":[\"null\",\"bytes\"],\"default\":null},"
+ + " {\"name\":\"reference\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"reference\",\"fields\":["
+ + " {\"name\":\"external_path\",\"type\":\"string\"},"
+ + " {\"name\":\"offset\",\"type\":\"long\"},"
+ + " {\"name\":\"length\",\"type\":\"long\"},"
+ + " {\"name\":\"managed\",\"type\":\"boolean\"}"
+ + " ]}],\"default\":null}"
+ + "]"
+ + "}";
Review Comment:
`BLOB_JSON` does not match the blob schema produced by `HoodieSchema.createBlob()`: it uses field name `storage_type` with type `string`, while `createBlob()` defines field `HoodieSchema.Blob.TYPE` (currently `"type"`) as an enum with symbols `[INLINE, OUT_OF_LINE]`. As written, the parse test can succeed while exercising a structurally-invalid blob schema, which weakens the validation coverage. Consider updating `BLOB_JSON` to mirror `createBlob()` (field names + enum vs string) and/or adding schema-structure validation in `HoodieSchema.fromAvroSchema` for blobs.
```suggestion
private static final String BLOB_JSON = HoodieSchema.createBlob().toString();
```
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1968,107 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
+ super.validate(schema);
+ if (schema.getType() != Schema.Type.RECORD) {
+ throw new IllegalArgumentException("Blob logical type can only be applied to RECORD schemas, got: " + schema.getType());
+ }
+ }
+ }
+
+ /**
+ * Factory for creating VariantLogicalType instances.
+ */
+ private static class BlogLogicalTypeFactory implements LogicalTypes.LogicalTypeFactory {
+ @Override
+ public LogicalType fromSchema(Schema schema) {
+ return BlobLogicalType.blob();
+ }
+
+ @Override
+ public String getTypeName() {
+ return BlobLogicalType.BLOB_LOGICAL_TYPE_NAME;
+ }
Review Comment:
There are a couple of blob logical-type naming/doc typos here: the factory class is named `BlogLogicalTypeFactory` (looks like it should be `BlobLogicalTypeFactory`), and its Javadoc says it creates `VariantLogicalType` instances even though itโs for blob. Renaming/fixing the Javadoc will make the new logical type easier to understand and avoid confusion later.
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaType.java:
##########
@@ -84,6 +84,8 @@ public void testComplexTypes() {
assertTrue(HoodieSchemaType.ARRAY.isComplex(), "ARRAY should be complex");
assertTrue(HoodieSchemaType.MAP.isComplex(), "MAP should be complex");
assertTrue(HoodieSchemaType.UNION.isComplex(), "UNION should be complex");
+ assertTrue(HoodieSchemaType.VARIANT.isComplex(), "VARIANT should be complex");
+ assertTrue(HoodieSchemaType.BLOB.isComplex(), "TIMESTAMP should be complex");
Review Comment:
The assertion message refers to TIMESTAMP, but the assertion is about BLOB being complex. This looks like a copy/paste typo and makes failures harder to interpret.
```suggestion
assertTrue(HoodieSchemaType.BLOB.isComplex(), "BLOB should be complex");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805255916
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +964,64 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (getType() == HoodieSchemaType.RECORD) {
Review Comment:
@the-other-tim-brown Just wondering if this copilot comment is valid? Are we allowing users to put blobs under VARIANT or should we be checking for this as well here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805314376
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##########
@@ -1689,4 +1690,208 @@ public void testVariantLogicalTypeSingleton() {
// Verify the logical type name
assertEquals("variant", instance1.getName());
}
+
+ // ==================== Blob Type Tests ====================
+
+ @Test
+ public void testCreateBlob() {
+ HoodieSchema.Blob blob = HoodieSchema.createBlob();
+
+ assertNotNull(blob);
+ assertEquals("blob", blob.getName());
+ assertEquals(HoodieSchemaType.BLOB, blob.getType());
+
+ // Verify storage_type field
+ Option<HoodieSchemaField> storageTypeOpt = blob.getField("type");
+ assertTrue(storageTypeOpt.isPresent());
+ HoodieSchemaField storageTypeField = storageTypeOpt.get();
+ assertEquals(HoodieSchemaType.ENUM, storageTypeField.schema().getType());
+ assertEquals(Arrays.asList("INLINE", "OUT_OF_LINE"), storageTypeField.schema().getEnumSymbols());
Review Comment:
The spec is `OUT_OF_LINE` https://github.com/apache/hudi/blob/master/rfc/rfc-100/rfc-100.md?plain=1#L102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3899102721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805264430
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +964,64 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (getType() == HoodieSchemaType.RECORD) {
Review Comment:
I just pushed an update to take this into account. I am going to also going to add checks to avoid blobs in variants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805296810
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##########
@@ -1689,4 +1690,208 @@ public void testVariantLogicalTypeSingleton() {
// Verify the logical type name
assertEquals("variant", instance1.getName());
}
+
+ // ==================== Blob Type Tests ====================
+
+ @Test
+ public void testCreateBlob() {
+ HoodieSchema.Blob blob = HoodieSchema.createBlob();
+
+ assertNotNull(blob);
+ assertEquals("blob", blob.getName());
+ assertEquals(HoodieSchemaType.BLOB, blob.getType());
+
+ // Verify storage_type field
+ Option<HoodieSchemaField> storageTypeOpt = blob.getField("type");
+ assertTrue(storageTypeOpt.isPresent());
+ HoodieSchemaField storageTypeField = storageTypeOpt.get();
+ assertEquals(HoodieSchemaType.ENUM, storageTypeField.schema().getType());
+ assertEquals(Arrays.asList("INLINE", "OUT_OF_LINE"), storageTypeField.schema().getEnumSymbols());
Review Comment:
Shouldn't the schema be either `INLINE` or `OUTLINE`?
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##########
@@ -1689,4 +1690,208 @@ public void testVariantLogicalTypeSingleton() {
// Verify the logical type name
assertEquals("variant", instance1.getName());
}
+
+ // ==================== Blob Type Tests ====================
+
+ @Test
+ public void testCreateBlob() {
+ HoodieSchema.Blob blob = HoodieSchema.createBlob();
+
+ assertNotNull(blob);
+ assertEquals("blob", blob.getName());
+ assertEquals(HoodieSchemaType.BLOB, blob.getType());
+
+ // Verify storage_type field
+ Option<HoodieSchemaField> storageTypeOpt = blob.getField("type");
+ assertTrue(storageTypeOpt.isPresent());
+ HoodieSchemaField storageTypeField = storageTypeOpt.get();
+ assertEquals(HoodieSchemaType.ENUM, storageTypeField.schema().getType());
+ assertEquals(Arrays.asList("INLINE", "OUT_OF_LINE"), storageTypeField.schema().getEnumSymbols());
Review Comment:
Shouldn't the schema be either `INLINE` or `OUT_OF_LINE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3891490227
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8525306001c7bd7d8baa807cc5398fbadb0d5bc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2828089962
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2047,14 +2047,15 @@ private static boolean isColumnTypeSupportedV1(HoodieSchema schema, Option<Hoodi
// HUDI-8585 will add support for BYTES and FIXED
return type != HoodieSchemaType.RECORD && type != HoodieSchemaType.ARRAY && type != HoodieSchemaType.MAP
&& type != HoodieSchemaType.ENUM && type != HoodieSchemaType.BYTES && type != HoodieSchemaType.FIXED
- && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
+ && type != HoodieSchemaType.BLOB && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
Review Comment:
What is the fix here? Generally I don't modify comments if they are not related to the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3891397236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d56bb1c0780e0c84c857caa8e385db8ba8e220d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976)
* f8525306001c7bd7d8baa807cc5398fbadb0d5bc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3909915617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* 575786e8dd1ad9f127d358eaaf66b7b666f57db8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2829704175
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2047,14 +2047,15 @@ private static boolean isColumnTypeSupportedV1(HoodieSchema schema, Option<Hoodi
// HUDI-8585 will add support for BYTES and FIXED
return type != HoodieSchemaType.RECORD && type != HoodieSchemaType.ARRAY && type != HoodieSchemaType.MAP
&& type != HoodieSchemaType.ENUM && type != HoodieSchemaType.BYTES && type != HoodieSchemaType.FIXED
- && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
+ && type != HoodieSchemaType.BLOB && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
Review Comment:
@the-other-tim-brown the comment is talking about just decimal. its related to the change where the line now also does stuff for BLOB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3874776408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3891824983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8525306001c7bd7d8baa807cc5398fbadb0d5bc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988)
* d7c1cf836715ff97315bfc765efac3624a00ec30 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3886480234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863)
* 074d15b66ad8e8bdbb2f2dee9a947eeb944180ed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942)
* 551d460cb256b7735dea2e2d4a788b60469143fe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3886920923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 074d15b66ad8e8bdbb2f2dee9a947eeb944180ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942)
* 551d460cb256b7735dea2e2d4a788b60469143fe Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805109551
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaType.java:
##########
@@ -156,6 +161,8 @@ public static HoodieSchemaType fromAvro(Schema avroSchema) {
return UUID;
} else if (logicalType instanceof VariantLogicalType) {
return VARIANT;
+ } else if (logicalType == HoodieSchema.BlobLogicalType.blob()) {
Review Comment:
The logical type is registered in the static block so this should not be an issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898356569
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
* 1d06141c80411593418964d6a86e7c1c91630fda Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3886545117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 074d15b66ad8e8bdbb2f2dee9a947eeb944180ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942)
* 551d460cb256b7735dea2e2d4a788b60469143fe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898339370
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
* 1d06141c80411593418964d6a86e7c1c91630fda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805275091
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1968,107 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
Review Comment:
Should we be doing any other validation here on the schema? I am assuming this is the avro model reference, so just curious if we need to check any other property at this point?
```
{
"type": "record",
"name": "Blob",
"fields": [
{"name": "type", "type": "enum", "symbols": ["INLINE", "OUT_OF_LINE"]},
{"name": "data", "type": ["null", "bytes"]},
{"name": "reference", "type": ["null", {
"type": "record",
"name": "BlobReference",
"fields": [
{"name": "external_path", "type": "string"},
{"name": "offset", "type": "long"},
{"name": "length", "type": "long"},
{"name": "managed", "type": "boolean"}
]
}]}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898550448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
* 1d06141c80411593418964d6a86e7c1c91630fda Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020)
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862233114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
}, {
"hash" : "466ff742031912b97bb660a31944a7f9ff359275",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "466ff742031912b97bb660a31944a7f9ff359275",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777)
* 466ff742031912b97bb660a31944a7f9ff359275 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2828109858
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +963,86 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.ARRAY) {
+ return getElementType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.MAP) {
+ return getValueType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (hasFields()) {
+ return getFields().stream().anyMatch(field -> field.schema().containsBlobType());
+ }
+ return false;
+ }
+
+ /**
+ * A convenience method to check if the current field represents a blob type.
+ * This checks if the current schema is a BLOB or if it is an ARRAY or MAP whose element or value type is a BLOB, respectively.
+ * It does not check for BLOB types nested within unions or record fields.
+ * @return true if the current schema is a BLOB or an ARRAY/MAP of BLOBs, false otherwise
+ */
+ public boolean isBlobField() {
+ HoodieSchema nonNullSchema = getNonNullType();
+ HoodieSchemaType nonNullSchemaType = nonNullSchema.getType();
+ return nonNullSchemaType == HoodieSchemaType.BLOB
+ || (nonNullSchemaType == HoodieSchemaType.ARRAY && nonNullSchema.getElementType().getNonNullType().getType() == HoodieSchemaType.BLOB)
+ || (nonNullSchemaType == HoodieSchemaType.MAP && nonNullSchema.getValueType().getNonNullType().getType() == HoodieSchemaType.BLOB);
+ }
+
+ /**
+ * Validates that the schema does not contain variants with shredded blob types.
+ * This method recursively traverses the schema tree to check for invalid structures.
+ *
+ * @param schema the schema to validate
+ * @throws HoodieSchemaException if the schema contains arrays or maps with blob types
+ */
+ private static void validateNoBlobsInVariant(HoodieSchema schema) {
+ if (schema == null) {
+ return;
+ }
+
+ HoodieSchemaType type = schema.getType();
+
+ switch (type) {
+ case ARRAY:
+ HoodieSchema elementType = schema.getElementType();
+ validateNoBlobsInVariant(elementType);
+ break;
+ case MAP:
+ HoodieSchema valueType = schema.getValueType();
+ validateNoBlobsInVariant(valueType);
+ break;
+ case VARIANT:
+ HoodieSchema.Variant variantSchema = (HoodieSchema.Variant) schema;
+ variantSchema.getTypedValueField().ifPresent(typedValueField -> {
+ if (typedValueField.getNonNullType().containsBlobType()) {
+ throw new HoodieSchemaException("Variant typed_value field cannot be or contain a BLOB type");
+ }
+ });
+ break;
+ case RECORD:
+ // Validate all record fields
+ List<HoodieSchemaField> fields = schema.getFields();
+ for (HoodieSchemaField field : fields) {
+ validateNoBlobsInVariant(field.schema());
+ }
+ break;
+ case UNION:
+ // Validate all union types
+ List<HoodieSchema> types = schema.getTypes();
+ for (HoodieSchema unionType : types) {
+ validateNoBlobsInVariant(unionType);
+ }
+ break;
+ // For primitives, BLOB, ENUM, FIXED, NULL - no nested validation needed
+ default:
+ break;
+ }
+ }
Review Comment:
Now that this is limited to inspecting variants, I think I can make the validation happen at the variant logical type level which would make this more efficient. I will test this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2829724826
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2047,14 +2047,15 @@ private static boolean isColumnTypeSupportedV1(HoodieSchema schema, Option<Hoodi
// HUDI-8585 will add support for BYTES and FIXED
return type != HoodieSchemaType.RECORD && type != HoodieSchemaType.ARRAY && type != HoodieSchemaType.MAP
&& type != HoodieSchemaType.ENUM && type != HoodieSchemaType.BYTES && type != HoodieSchemaType.FIXED
- && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
+ && type != HoodieSchemaType.BLOB && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
Review Comment:
Ok so should it just be on its own line? Decimal is still backed by bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3930252967
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
}, {
"hash" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127",
"triggerID" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* bfbbf46b7473d4f34965b5a311418ff1c678ff6f UNKNOWN
* Unknown: [CANCELED](TBD)
* cae177ea0924da786dc0ab258cbfd11ea31a51a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12127)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805388502
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestSchemaConverters.scala:
##########
@@ -18,12 +18,15 @@
package org.apache.spark.sql.avro
import org.apache.hudi.avro.model.HoodieMetadataColumnStats
-import org.apache.hudi.common.schema.HoodieSchema
+import org.apache.hudi.common.schema.{HoodieSchema, HoodieSchemaField, HoodieSchemaType}
import org.apache.avro.JsonProperties
-import org.junit.jupiter.api.Assertions.assertEquals
+import org.apache.spark.sql.types.{DataTypes, MetadataBuilder, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
import org.junit.jupiter.api.Test
+import java.util
+
class TestSchemaConverters {
Review Comment:
This is more a general comment in terms of where the tests should be placed. Currently I noticed that all our testing for different types between Spark struct type and HoodieSchema are done in the following class here: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSchemaConversionUtils.scala#L34
I have also put my vector Schema related tests inside the above class as well: https://github.com/apache/hudi/pull/18190/changes#diff-70e28ca555d1f86b32e1c2965c9c5435da6e59a0ac89f2d1ff24a54fe5af1f6aR573
Im wondering then we should we either continue to put tests in TestHoodieSchemaConversionUtils or should we be having these new type tests in TestSchemaConverters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3887394401
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 551d460cb256b7735dea2e2d4a788b60469143fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898656736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
* 1d06141c80411593418964d6a86e7c1c91630fda Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020)
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3888700945
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
* d56bb1c0780e0c84c857caa8e385db8ba8e220d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3888986534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d56bb1c0780e0c84c857caa8e385db8ba8e220d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3863284468
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
}, {
"hash" : "466ff742031912b97bb660a31944a7f9ff359275",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "466ff742031912b97bb660a31944a7f9ff359275",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "e7b992361e1a04676e4e8432d48c5db85ccb58c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7b992361e1a04676e4e8432d48c5db85ccb58c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 466ff742031912b97bb660a31944a7f9ff359275 UNKNOWN
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
* e7b992361e1a04676e4e8432d48c5db85ccb58c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3909442262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2829714768
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1985,127 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
+ super.validate(schema);
+ if (schema.getType() != Schema.Type.RECORD) {
+ throw new IllegalArgumentException("Blob logical type can only be applied to RECORD schemas, got: " + schema.getType());
+ }
+ if (!schema.getFields().equals(HoodieSchema.Blob.BLOB_FIELDS)) {
+ throw new IllegalArgumentException("Blob logical type cannot be applied to schema: " + schema);
+ }
+ }
Review Comment:
Just flagging that we are relying on the underlying avro library to do the check. The flink check was handwritten? its a suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3891412664
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d56bb1c0780e0c84c857caa8e385db8ba8e220d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976)
* f8525306001c7bd7d8baa807cc5398fbadb0d5bc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3892469921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862109854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862356249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
}, {
"hash" : "466ff742031912b97bb660a31944a7f9ff359275",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "466ff742031912b97bb660a31944a7f9ff359275",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e282c399fe212376a4cceb8930debfff858a113 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777)
* 466ff742031912b97bb660a31944a7f9ff359275 UNKNOWN
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3862664811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e282c399fe212376a4cceb8930debfff858a113",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11777",
"triggerID" : "2e282c399fe212376a4cceb8930debfff858a113",
"triggerType" : "PUSH"
}, {
"hash" : "466ff742031912b97bb660a31944a7f9ff359275",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "466ff742031912b97bb660a31944a7f9ff359275",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 466ff742031912b97bb660a31944a7f9ff359275 UNKNOWN
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3886442591
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863)
* 074d15b66ad8e8bdbb2f2dee9a947eeb944180ed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805350839
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1968,107 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
Review Comment:
Makes sense, added validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805125094
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -152,20 +159,28 @@ public static HoodieSchema fromAvroSchema(Schema avroSchema) {
if (avroSchema == null) {
return null;
}
+ HoodieSchema schema;
LogicalType logicalType = avroSchema.getLogicalType();
if (logicalType != null) {
if (logicalType instanceof LogicalTypes.Decimal) {
- return new HoodieSchema.Decimal(avroSchema);
+ schema = new HoodieSchema.Decimal(avroSchema);
} else if (logicalType instanceof LogicalTypes.TimeMillis || logicalType instanceof LogicalTypes.TimeMicros) {
- return new HoodieSchema.Time(avroSchema);
+ schema = new HoodieSchema.Time(avroSchema);
} else if (logicalType instanceof LogicalTypes.TimestampMillis || logicalType instanceof LogicalTypes.TimestampMicros
|| logicalType instanceof LogicalTypes.LocalTimestampMillis || logicalType instanceof LogicalTypes.LocalTimestampMicros) {
- return new HoodieSchema.Timestamp(avroSchema);
+ schema = new HoodieSchema.Timestamp(avroSchema);
} else if (logicalType == VariantLogicalType.variant()) {
- return new HoodieSchema.Variant(avroSchema);
+ schema = new HoodieSchema.Variant(avroSchema);
+ } else if (logicalType == BlobLogicalType.blob()) {
Review Comment:
The logical type is registered in the static block so this should not be an issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805283311
##########
hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchema.java:
##########
@@ -1689,4 +1690,221 @@ public void testVariantLogicalTypeSingleton() {
// Verify the logical type name
assertEquals("variant", instance1.getName());
}
+
+ // ==================== Blob Type Tests ====================
Review Comment:
I think we can remove this comment right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3909505103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3909554296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* 575786e8dd1ad9f127d358eaaf66b7b666f57db8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2827752859
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -79,20 +80,25 @@ object HoodieSparkSchemaConverters {
// Complex types
case ArrayType(elementType, containsNull) =>
- val elementSchema = toHoodieType(elementType, containsNull, recordName, nameSpace)
+ val elementSchema = toHoodieType(elementType, containsNull, recordName, nameSpace, metadata)
HoodieSchema.createArray(elementSchema)
case MapType(StringType, valueType, valueContainsNull) =>
- val valueSchema = toHoodieType(valueType, valueContainsNull, recordName, nameSpace)
+ val valueSchema = toHoodieType(valueType, valueContainsNull, recordName, nameSpace, metadata)
HoodieSchema.createMap(valueSchema)
+ case st: StructType if metadata.contains(HoodieSchema.TYPE_METADATA_FIELD) &&
Review Comment:
๐
**NIT**
rename `st` => `blobStruct`
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +963,86 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.ARRAY) {
+ return getElementType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.MAP) {
+ return getValueType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (hasFields()) {
+ return getFields().stream().anyMatch(field -> field.schema().containsBlobType());
+ }
+ return false;
+ }
+
+ /**
+ * A convenience method to check if the current field represents a blob type.
+ * This checks if the current schema is a BLOB or if it is an ARRAY or MAP whose element or value type is a BLOB, respectively.
+ * It does not check for BLOB types nested within unions or record fields.
+ * @return true if the current schema is a BLOB or an ARRAY/MAP of BLOBs, false otherwise
+ */
+ public boolean isBlobField() {
+ HoodieSchema nonNullSchema = getNonNullType();
+ HoodieSchemaType nonNullSchemaType = nonNullSchema.getType();
+ return nonNullSchemaType == HoodieSchemaType.BLOB
+ || (nonNullSchemaType == HoodieSchemaType.ARRAY && nonNullSchema.getElementType().getNonNullType().getType() == HoodieSchemaType.BLOB)
+ || (nonNullSchemaType == HoodieSchemaType.MAP && nonNullSchema.getValueType().getNonNullType().getType() == HoodieSchemaType.BLOB);
+ }
+
+ /**
+ * Validates that the schema does not contain variants with shredded blob types.
+ * This method recursively traverses the schema tree to check for invalid structures.
+ *
+ * @param schema the schema to validate
+ * @throws HoodieSchemaException if the schema contains arrays or maps with blob types
+ */
+ private static void validateNoBlobsInVariant(HoodieSchema schema) {
Review Comment:
๐
**NIT**
rename to. `validateNoBlobsInsideVariant` .. First time, I read it as `invariant`
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -190,11 +196,21 @@ object HoodieSparkSchemaConverters {
val newRecordNames = existingRecordNames + fullName
val fields = hoodieSchema.getFields.asScala.map { f =>
val schemaType = toSqlTypeHelper(f.schema(), newRecordNames)
- val metadata = if (f.doc().isPresent && !f.doc().get().isEmpty) {
+ val commentMetadata = if (f.doc().isPresent && !f.doc().get().isEmpty) {
new MetadataBuilder().putString("comment", f.doc().get()).build()
} else {
Metadata.empty
}
+ val fieldSchema = f.getNonNullSchema
+ val metadata = if (fieldSchema.isBlobField) {
+ // Mark blob fields with metadata for identification
+ new MetadataBuilder()
+ .withMetadata(commentMetadata)
+ .putString(HoodieSchema.TYPE_METADATA_FIELD, HoodieSchemaType.BLOB.name())
+ .build()
+ } else {
+ commentMetadata
+ }
StructField(f.name(), schemaType.dataType, schemaType.nullable, metadata)
}
SchemaType(StructType(fields.toSeq), nullable = false)
Review Comment:
๐ฌ **SUGGESTION: This works as long as blobs are always nested inside a parent record in practice. i.e not in `catalystType` directly is never a BLOB. This is an acceptable limitation โ just worth documenting explicitly.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1985,127 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
+ super.validate(schema);
+ if (schema.getType() != Schema.Type.RECORD) {
+ throw new IllegalArgumentException("Blob logical type can only be applied to RECORD schemas, got: " + schema.getType());
+ }
+ if (!schema.getFields().equals(HoodieSchema.Blob.BLOB_FIELDS)) {
+ throw new IllegalArgumentException("Blob logical type cannot be applied to schema: " + schema);
+ }
+ }
Review Comment:
๐ฌ **SUGGESTION**:
```java
if (!schema.getFields().equals(HoodieSchema.Blob.BLOB_FIELDS)) {
```
Here we are relying on order and field equals() from avro. Should we explictly compare the individual fields like we did for Spark/Flink converters?
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +963,86 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.ARRAY) {
+ return getElementType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.MAP) {
+ return getValueType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (hasFields()) {
+ return getFields().stream().anyMatch(field -> field.schema().containsBlobType());
+ }
+ return false;
+ }
+
+ /**
+ * A convenience method to check if the current field represents a blob type.
+ * This checks if the current schema is a BLOB or if it is an ARRAY or MAP whose element or value type is a BLOB, respectively.
+ * It does not check for BLOB types nested within unions or record fields.
+ * @return true if the current schema is a BLOB or an ARRAY/MAP of BLOBs, false otherwise
+ */
+ public boolean isBlobField() {
+ HoodieSchema nonNullSchema = getNonNullType();
+ HoodieSchemaType nonNullSchemaType = nonNullSchema.getType();
+ return nonNullSchemaType == HoodieSchemaType.BLOB
+ || (nonNullSchemaType == HoodieSchemaType.ARRAY && nonNullSchema.getElementType().getNonNullType().getType() == HoodieSchemaType.BLOB)
+ || (nonNullSchemaType == HoodieSchemaType.MAP && nonNullSchema.getValueType().getNonNullType().getType() == HoodieSchemaType.BLOB);
+ }
+
+ /**
+ * Validates that the schema does not contain variants with shredded blob types.
+ * This method recursively traverses the schema tree to check for invalid structures.
+ *
+ * @param schema the schema to validate
+ * @throws HoodieSchemaException if the schema contains arrays or maps with blob types
+ */
+ private static void validateNoBlobsInVariant(HoodieSchema schema) {
+ if (schema == null) {
+ return;
+ }
+
+ HoodieSchemaType type = schema.getType();
+
+ switch (type) {
+ case ARRAY:
+ HoodieSchema elementType = schema.getElementType();
+ validateNoBlobsInVariant(elementType);
+ break;
+ case MAP:
+ HoodieSchema valueType = schema.getValueType();
+ validateNoBlobsInVariant(valueType);
+ break;
+ case VARIANT:
+ HoodieSchema.Variant variantSchema = (HoodieSchema.Variant) schema;
+ variantSchema.getTypedValueField().ifPresent(typedValueField -> {
+ if (typedValueField.getNonNullType().containsBlobType()) {
+ throw new HoodieSchemaException("Variant typed_value field cannot be or contain a BLOB type");
+ }
+ });
+ break;
+ case RECORD:
+ // Validate all record fields
+ List<HoodieSchemaField> fields = schema.getFields();
+ for (HoodieSchemaField field : fields) {
+ validateNoBlobsInVariant(field.schema());
+ }
+ break;
+ case UNION:
+ // Validate all union types
+ List<HoodieSchema> types = schema.getTypes();
+ for (HoodieSchema unionType : types) {
+ validateNoBlobsInVariant(unionType);
+ }
+ break;
+ // For primitives, BLOB, ENUM, FIXED, NULL - no nested validation needed
+ default:
+ break;
+ }
+ }
Review Comment:
๐
**NIT: `validateNoBlobsInVariant` is called on every `fromAvroSchema` call**
This adds a recursive traversal on every schema parse. For schemas without variants this traversal is wasted work. Since this validation only matters for variant+blob combinations, it could be gated on whether the schema actually contains a variant or piggybacked on any other traversal already done on this nature? Performance impact is negligible for smaller/common schemas, but worth thinking for large schemas with many nested types.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2047,14 +2047,15 @@ private static boolean isColumnTypeSupportedV1(HoodieSchema schema, Option<Hoodi
// HUDI-8585 will add support for BYTES and FIXED
return type != HoodieSchemaType.RECORD && type != HoodieSchemaType.ARRAY && type != HoodieSchemaType.MAP
&& type != HoodieSchemaType.ENUM && type != HoodieSchemaType.BYTES && type != HoodieSchemaType.FIXED
- && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
+ && type != HoodieSchemaType.BLOB && type != HoodieSchemaType.DECIMAL; // DECIMAL's underlying type is BYTES
Review Comment:
๐
**NIT** fix or remove comment
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestSchemaConverters.scala:
##########
@@ -18,12 +18,15 @@
package org.apache.spark.sql.avro
import org.apache.hudi.avro.model.HoodieMetadataColumnStats
-import org.apache.hudi.common.schema.HoodieSchema
+import org.apache.hudi.common.schema.{HoodieSchema, HoodieSchemaField, HoodieSchemaType}
import org.apache.avro.JsonProperties
-import org.junit.jupiter.api.Assertions.assertEquals
+import org.apache.spark.sql.types.{DataTypes, MetadataBuilder, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
import org.junit.jupiter.api.Test
+import java.util
+
class TestSchemaConverters {
Review Comment:
We should home those tests in right places. Need not happen in this PR per se.
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -630,6 +648,10 @@ public static HoodieSchema.Variant createVariantShredded(String name, String nam
return new HoodieSchema.Variant(recordSchema);
}
+ public static HoodieSchema.Blob createBlob() {
+ return new HoodieSchema.Blob(Blob.DEFAULT_NAME);
+ }
Review Comment:
โ ๏ธ **IMPORTANT**
Does using the DEFAULT_NAME work when there are multiple BLOB fields in the schema.? or do we need a way to pass in the name as we do for variant/decimal
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +963,86 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.ARRAY) {
+ return getElementType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.MAP) {
+ return getValueType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (hasFields()) {
+ return getFields().stream().anyMatch(field -> field.schema().containsBlobType());
+ }
+ return false;
+ }
+
+ /**
+ * A convenience method to check if the current field represents a blob type.
+ * This checks if the current schema is a BLOB or if it is an ARRAY or MAP whose element or value type is a BLOB, respectively.
+ * It does not check for BLOB types nested within unions or record fields.
+ * @return true if the current schema is a BLOB or an ARRAY/MAP of BLOBs, false otherwise
+ */
+ public boolean isBlobField() {
+ HoodieSchema nonNullSchema = getNonNullType();
+ HoodieSchemaType nonNullSchemaType = nonNullSchema.getType();
+ return nonNullSchemaType == HoodieSchemaType.BLOB
+ || (nonNullSchemaType == HoodieSchemaType.ARRAY && nonNullSchema.getElementType().getNonNullType().getType() == HoodieSchemaType.BLOB)
+ || (nonNullSchemaType == HoodieSchemaType.MAP && nonNullSchema.getValueType().getNonNullType().getType() == HoodieSchemaType.BLOB);
+ }
+
+ /**
+ * Validates that the schema does not contain variants with shredded blob types.
+ * This method recursively traverses the schema tree to check for invalid structures.
+ *
+ * @param schema the schema to validate
+ * @throws HoodieSchemaException if the schema contains arrays or maps with blob types
+ */
+ private static void validateNoBlobsInVariant(HoodieSchema schema) {
+ if (schema == null) {
+ return;
+ }
+
+ HoodieSchemaType type = schema.getType();
+
+ switch (type) {
+ case ARRAY:
+ HoodieSchema elementType = schema.getElementType();
+ validateNoBlobsInVariant(elementType);
+ break;
+ case MAP:
+ HoodieSchema valueType = schema.getValueType();
+ validateNoBlobsInVariant(valueType);
+ break;
+ case VARIANT:
+ HoodieSchema.Variant variantSchema = (HoodieSchema.Variant) schema;
+ variantSchema.getTypedValueField().ifPresent(typedValueField -> {
+ if (typedValueField.getNonNullType().containsBlobType()) {
+ throw new HoodieSchemaException("Variant typed_value field cannot be or contain a BLOB type");
+ }
+ });
+ break;
+ case RECORD:
+ // Validate all record fields
+ List<HoodieSchemaField> fields = schema.getFields();
Review Comment:
๐
**NIT**
Can we do 1 line lambdas like `schema.getFields().forEach(field -> validateNoBlobsInVariant(field.schema()));` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2828115057
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -1883,6 +1985,127 @@ public int hashCode() {
}
}
+ static class BlobLogicalType extends LogicalType {
+
+ private static final String BLOB_LOGICAL_TYPE_NAME = "blob";
+ // Eager initialization of singleton
+ private static final BlobLogicalType INSTANCE = new BlobLogicalType();
+
+ private BlobLogicalType() {
+ super(BlobLogicalType.BLOB_LOGICAL_TYPE_NAME);
+ }
+
+ public static BlobLogicalType blob() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void validate(Schema schema) {
+ super.validate(schema);
+ if (schema.getType() != Schema.Type.RECORD) {
+ throw new IllegalArgumentException("Blob logical type can only be applied to RECORD schemas, got: " + schema.getType());
+ }
+ if (!schema.getFields().equals(HoodieSchema.Blob.BLOB_FIELDS)) {
+ throw new IllegalArgumentException("Blob logical type cannot be applied to schema: " + schema);
+ }
+ }
Review Comment:
The spark check is `structType.equals(expectedBlobStructType)` so I am unsure what this comment is trying to highlight. Is there a concern around being too strict on the equality check here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2828243326
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +963,86 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.ARRAY) {
+ return getElementType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.MAP) {
+ return getValueType().containsBlobType();
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (hasFields()) {
+ return getFields().stream().anyMatch(field -> field.schema().containsBlobType());
+ }
+ return false;
+ }
+
+ /**
+ * A convenience method to check if the current field represents a blob type.
+ * This checks if the current schema is a BLOB or if it is an ARRAY or MAP whose element or value type is a BLOB, respectively.
+ * It does not check for BLOB types nested within unions or record fields.
+ * @return true if the current schema is a BLOB or an ARRAY/MAP of BLOBs, false otherwise
+ */
+ public boolean isBlobField() {
+ HoodieSchema nonNullSchema = getNonNullType();
+ HoodieSchemaType nonNullSchemaType = nonNullSchema.getType();
+ return nonNullSchemaType == HoodieSchemaType.BLOB
+ || (nonNullSchemaType == HoodieSchemaType.ARRAY && nonNullSchema.getElementType().getNonNullType().getType() == HoodieSchemaType.BLOB)
+ || (nonNullSchemaType == HoodieSchemaType.MAP && nonNullSchema.getValueType().getNonNullType().getType() == HoodieSchemaType.BLOB);
+ }
+
+ /**
+ * Validates that the schema does not contain variants with shredded blob types.
+ * This method recursively traverses the schema tree to check for invalid structures.
+ *
+ * @param schema the schema to validate
+ * @throws HoodieSchemaException if the schema contains arrays or maps with blob types
+ */
+ private static void validateNoBlobsInVariant(HoodieSchema schema) {
+ if (schema == null) {
+ return;
+ }
+
+ HoodieSchemaType type = schema.getType();
+
+ switch (type) {
+ case ARRAY:
+ HoodieSchema elementType = schema.getElementType();
+ validateNoBlobsInVariant(elementType);
+ break;
+ case MAP:
+ HoodieSchema valueType = schema.getValueType();
+ validateNoBlobsInVariant(valueType);
+ break;
+ case VARIANT:
+ HoodieSchema.Variant variantSchema = (HoodieSchema.Variant) schema;
+ variantSchema.getTypedValueField().ifPresent(typedValueField -> {
+ if (typedValueField.getNonNullType().containsBlobType()) {
+ throw new HoodieSchemaException("Variant typed_value field cannot be or contain a BLOB type");
+ }
+ });
+ break;
+ case RECORD:
+ // Validate all record fields
+ List<HoodieSchemaField> fields = schema.getFields();
+ for (HoodieSchemaField field : fields) {
+ validateNoBlobsInVariant(field.schema());
+ }
+ break;
+ case UNION:
+ // Validate all union types
+ List<HoodieSchema> types = schema.getTypes();
+ for (HoodieSchema unionType : types) {
+ validateNoBlobsInVariant(unionType);
+ }
+ break;
+ // For primitives, BLOB, ENUM, FIXED, NULL - no nested validation needed
+ default:
+ break;
+ }
+ }
Review Comment:
It turns out that the avro parsing will not throw errors on invalid logical types. The error will not be thrown on parsing but can be thrown when trying to access the field or when creating the field programatically.
I think that this case of blob inside of variant will be unlikely so I think throwing at runtime is a fair tradeoff here to avoid traversing the schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3927746232
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
}, {
"hash" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* 575786e8dd1ad9f127d358eaaf66b7b666f57db8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074)
* bfbbf46b7473d4f34965b5a311418ff1c678ff6f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2829710594
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -630,6 +648,10 @@ public static HoodieSchema.Variant createVariantShredded(String name, String nam
return new HoodieSchema.Variant(recordSchema);
}
+ public static HoodieSchema.Blob createBlob() {
+ return new HoodieSchema.Blob(Blob.DEFAULT_NAME);
+ }
Review Comment:
> the schema of the same name can be reused
This is my main concern. not user flexibility on naming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3874757783
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1938b93b9abb39c3cf913e20753055e829b534ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805255916
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchema.java:
##########
@@ -941,6 +964,64 @@ public HoodieSchema getNonNullType() {
return HoodieSchema.createUnion(nonNullTypes);
}
+ boolean containsBlobType() {
+ if (getType() == HoodieSchemaType.BLOB) {
+ return true;
+ } else if (getType() == HoodieSchemaType.UNION) {
+ return getTypes().stream().anyMatch(HoodieSchema::containsBlobType);
+ } else if (getType() == HoodieSchemaType.RECORD) {
Review Comment:
@the-other-tim-brown Just wondering if this copilot comment is valid? Are we allowing users to put blobs under VARIANT or MAP or should this be checking for those complex types as well here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3898473341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
* 1d06141c80411593418964d6a86e7c1c91630fda Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020)
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #18108:
URL: https://github.com/apache/hudi/pull/18108#discussion_r2805402947
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestSchemaConverters.scala:
##########
@@ -18,12 +18,15 @@
package org.apache.spark.sql.avro
import org.apache.hudi.avro.model.HoodieMetadataColumnStats
-import org.apache.hudi.common.schema.HoodieSchema
+import org.apache.hudi.common.schema.{HoodieSchema, HoodieSchemaField, HoodieSchemaType}
import org.apache.avro.JsonProperties
-import org.junit.jupiter.api.Assertions.assertEquals
+import org.apache.spark.sql.types.{DataTypes, MetadataBuilder, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
import org.junit.jupiter.api.Test
+import java.util
+
class TestSchemaConverters {
Review Comment:
I wasn't aware of the other test class. I was just looking for where `HoodieSparkSchemaConverters` was already referenced in tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3888253351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 551d460cb256b7735dea2e2d4a788b60469143fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945)
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3888694469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b78d8b5405c132669a1623cd1f9af53bff1edfa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783)
* d56bb1c0780e0c84c857caa8e385db8ba8e220d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3891839839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f8525306001c7bd7d8baa807cc5398fbadb0d5bc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988)
* d7c1cf836715ff97315bfc765efac3624a00ec30 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3909530013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 36dc80c4ead35769b1e9f626c32882197d37fe99 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023)
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* 575786e8dd1ad9f127d358eaaf66b7b666f57db8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3929092512
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(blob): Blob schema definition [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18108:
URL: https://github.com/apache/hudi/pull/18108#issuecomment-3930213515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11863",
"triggerID" : "1938b93b9abb39c3cf913e20753055e829b534ff",
"triggerType" : "PUSH"
}, {
"hash" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11942",
"triggerID" : "074d15b66ad8e8bdbb2f2dee9a947eeb944180ed",
"triggerType" : "PUSH"
}, {
"hash" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11945",
"triggerID" : "551d460cb256b7735dea2e2d4a788b60469143fe",
"triggerType" : "PUSH"
}, {
"hash" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11783",
"triggerID" : "4b78d8b5405c132669a1623cd1f9af53bff1edfa",
"triggerType" : "PUSH"
}, {
"hash" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11976",
"triggerID" : "d56bb1c0780e0c84c857caa8e385db8ba8e220d3",
"triggerType" : "PUSH"
}, {
"hash" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11988",
"triggerID" : "f8525306001c7bd7d8baa807cc5398fbadb0d5bc",
"triggerType" : "PUSH"
}, {
"hash" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11990",
"triggerID" : "d7c1cf836715ff97315bfc765efac3624a00ec30",
"triggerType" : "PUSH"
}, {
"hash" : "1d06141c80411593418964d6a86e7c1c91630fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12020",
"triggerID" : "1d06141c80411593418964d6a86e7c1c91630fda",
"triggerType" : "PUSH"
}, {
"hash" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02d82154c8d9f7482ae4f917dd22c9dcc92415bf",
"triggerType" : "PUSH"
}, {
"hash" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12023",
"triggerID" : "36dc80c4ead35769b1e9f626c32882197d37fe99",
"triggerType" : "PUSH"
}, {
"hash" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e93ac79c3634351c0fb49ef7e38c39169fd4045",
"triggerType" : "PUSH"
}, {
"hash" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6",
"triggerType" : "PUSH"
}, {
"hash" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12074",
"triggerID" : "575786e8dd1ad9f127d358eaaf66b7b666f57db8",
"triggerType" : "PUSH"
}, {
"hash" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfbbf46b7473d4f34965b5a311418ff1c678ff6f",
"triggerType" : "PUSH"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "3929092512",
"triggerType" : "MANUAL"
}, {
"hash" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cae177ea0924da786dc0ab258cbfd11ea31a51a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 02d82154c8d9f7482ae4f917dd22c9dcc92415bf UNKNOWN
* 5e93ac79c3634351c0fb49ef7e38c39169fd4045 UNKNOWN
* f6f4a2f9a3778ca4a12eafb4fdebb4387ddcbfd6 UNKNOWN
* bfbbf46b7473d4f34965b5a311418ff1c678ff6f UNKNOWN
* Unknown: [CANCELED](TBD)
* cae177ea0924da786dc0ab258cbfd11ea31a51a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org