You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2025/12/11 19:39:59 UTC
[PR] Rahil/spark migration reader [hudi]
rahil-c opened a new pull request, #17573:
URL: https://github.com/apache/hudi/pull/17573
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631889665
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriterWithTestFormat.scala:
##########
@@ -268,8 +268,8 @@ class TestHoodieSparkSqlWriterWithTestFormat extends HoodieSparkWriterTestBase {
.updated(HoodieTableConfig.TABLE_FORMAT.key, "test-format")
// generate the inserts
- val schema = DataSourceTestUtils.getStructTypeExampleSchema
- val structType = AvroConversionUtils.convertAvroSchemaToStructType(schema)
+ val schema = DataSourceTestUtils.getStructTypeExampleHoodieSchema
Review Comment:
makes sense will do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631955181
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -49,17 +51,17 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializerInternal(rootType: HoodieSchema,
rootCatalystType: DataType,
positionalFieldMatch: Boolean,
datetimeRebaseSpec: RebaseSpec,
filters: StructFilters) {
- def this(rootAvroType: Schema,
+ def this(rootType: HoodieSchema,
Review Comment:
Ok i will look into removing these set of changes of passing the HoodieSchema to the actual AvroSerializer and AvroDeserializer classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672076450
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
* 18706cf60f015018b214ea78997b17894198021a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662980993
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303)
* 24dda257806ad8dae89de72faf95f04e02242399 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662976909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303)
* 24dda257806ad8dae89de72faf95f04e02242399 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643714796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 204f6219585adb778e51680eaf3bcedeb8e8d178 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143)
* 8b3fff68c0346d4464e75e29b62578dca730043a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3652258663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6a6deb8869158c0d0e18fc8e48d05009656c952 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226)
* 790babc0c83f242c3265bdbb5866e5b71230fb25 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620864967
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -841,13 +840,13 @@ object HoodieBaseRelation extends SparkAdapterSupport {
val requiredRowSchema = requiredDataSchema.structTypeSchema
// NOTE: Schema has to be parsed at this point, since Avro's [[Schema]] aren't serializable
// to be passed from driver to executor
- val requiredAvroSchema = new Schema.Parser().parse(requiredDataSchema.avroSchemaStr)
- val avroToRowConverter = AvroConversionUtils.createAvroToInternalRowConverter(requiredAvroSchema, requiredRowSchema)
+ val requiredSchema = HoodieSchema.parse(requiredDataSchema.avroSchemaStr)
+ //TODO to fill impl for this
Review Comment:
This comment actually can be removed as its been addressed. Since we now have added a util `HoodieSchemaConversionUtils.createGenericRecordToInternalRowConverter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3650184199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a49e78d78fb9a210b6405236482916de1bc1a066 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3647855942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
* b766aed2cb16e04408b1fba1a83f8673177836be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184)
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631926898
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
Review Comment:
Sure we can add another api in `TableSchemaResolver` that will basically just delegate to the underlying getTableAvroSchemaFromDataFile? This is pattern we have for example with `TableSchemaResolver #getTableSchema`
<img width="856" height="305" alt="Screenshot 2025-12-18 at 9 18 16 AM" src="https://github.com/user-attachments/assets/92374812-c8f2-4cdb-97b0-fd67d14c8e5f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632634129
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
Review Comment:
Followed @yihua recommendation here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632384613
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -79,6 +79,24 @@ public static HoodieSchema createHoodieWriteSchema(String schema, boolean withOp
return HoodieSchema.fromAvroSchema(avroSchema);
}
+ /**
+ * Creates a write schema for Hudi operations, adding necessary metadata fields.
+ * This is equivalent to HoodieAvroUtils.createHoodieWriteSchema() but operates on HoodieSchema.
+ *
+ * @param schema the base HoodieSchema
+ * @param withOperationField whether to include operation metadata field
+ * @return HoodieSchema configured for write operations with metadata fields added
+ * @throws IllegalArgumentException if schema is null
+ * @since 1.2.0
+ */
+ public static HoodieSchema createHoodieWriteSchema(HoodieSchema schema, boolean withOperationField) {
+ ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
+
+ // Convert to Avro, delegate to existing utility, convert back
+ Schema avroSchema = HoodieAvroUtils.addMetadataFields(schema.toAvroSchema(), withOperationField);
Review Comment:
@yihua requested to create a followup github task(not related to this pr) for having the utils be part of the actual `HoodieSchema` class instead of maintaining `HoodieSchemaUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3656378440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 790babc0c83f242c3265bdbb5866e5b71230fb25 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229)
* 3db175289bb9a54c723b7ea5972c46c81300c0fe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620371931
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -841,13 +840,13 @@ object HoodieBaseRelation extends SparkAdapterSupport {
val requiredRowSchema = requiredDataSchema.structTypeSchema
// NOTE: Schema has to be parsed at this point, since Avro's [[Schema]] aren't serializable
// to be passed from driver to executor
- val requiredAvroSchema = new Schema.Parser().parse(requiredDataSchema.avroSchemaStr)
- val avroToRowConverter = AvroConversionUtils.createAvroToInternalRowConverter(requiredAvroSchema, requiredRowSchema)
+ val requiredSchema = HoodieSchema.parse(requiredDataSchema.avroSchemaStr)
+ //TODO to fill impl for this
Review Comment:
to remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3657931429
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
* 40f3c37722c93a9a37136f128ab5fc135feb34b9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257)
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620858866
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -58,7 +56,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SparkSession, SQLContext}
import org.apache.spark.sql.HoodieCatalystExpressionUtils.{convertToCatalystExpression, generateUnsafeProjection}
-import org.apache.spark.sql.avro.HoodieSparkAvroSchemaConverters
+import org.apache.spark.sql.avro.{HoodieSparkAvroSchemaConverters, HoodieSparkSchemaConverters}
Review Comment:
Would it be possible to instead have as a separate removal task similar to https://github.com/apache/hudi/issues/14282? We can have it a fast follow after this that way we can aim to land this large piece first, but let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3648551070
Discussed with @yihua on the PR, one suggestion was that for `AvroSerializer` and `deserilaizer`, we will keep most of the logic to be on Avro Schema as we anyway are serializing to avro constructs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3657810523
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
* 40f3c37722c93a9a37136f128ab5fc135feb34b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3659230827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e82c5dd3f3374e1806f3d358379f9f4e85f174a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662437027
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628726020
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -49,17 +51,17 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializerInternal(rootType: HoodieSchema,
rootCatalystType: DataType,
positionalFieldMatch: Boolean,
datetimeRebaseSpec: RebaseSpec,
filters: StructFilters) {
- def this(rootAvroType: Schema,
+ def this(rootType: HoodieSchema,
Review Comment:
@rahil-c I thought we want to keep the `AvroSerializer` and `AvroDeserializer` classes untouched and let Spark adapter class to pass in the Avro schema, to limit the less-valuable changes?
```
override def createAvroSerializer(rootCatalystType: DataType, rootType: HoodieSchema, nullable: Boolean): HoodieAvroSerializer =
new HoodieSpark3_3AvroSerializer(rootCatalystType, rootType.toAvroSchema, nullable)
override def createAvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType): HoodieAvroDeserializer =
new HoodieSpark3_3AvroDeserializer(rootType.toAvroSchema, rootCatalystType)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675096278
I've addressed the comments, but i do not have permissions to push to @rahil-c's branch.
If anyone has permissions, you can apply the changes using the patch below:
[Address_styling_comments_1.patch](https://github.com/user-attachments/files/24258742/Address_styling_comments_1.patch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633470860
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/ColumnStatIndexTestBase.scala:
##########
@@ -36,7 +36,7 @@ import org.apache.hudi.storage.StoragePath
import org.apache.hudi.storage.hadoop.HadoopStorageConfiguration
import org.apache.hudi.testutils.{HoodieSparkClientTestBase, LogFileColStatsTestUtil}
-import org.apache.avro.Schema
+import org.apache.avro.{Schema}
Review Comment:
nit: brackets are redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643514351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 204f6219585adb778e51680eaf3bcedeb8e8d178 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3648144814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617423850
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableChangeColumnCommand.scala:
##########
@@ -86,10 +85,10 @@ case class AlterHoodieTableChangeColumnCommand(
Seq.empty[Row]
}
- private def validateSchema(newSchema: Schema, metaClient: HoodieTableMetaClient): Unit = {
+ private def validateSchema(newSchema: HoodieSchema, metaClient: HoodieTableMetaClient): Unit = {
val schemaUtil = new TableSchemaResolver(metaClient)
- val tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchema(false))
- if (!AvroSchemaUtils.isSchemaCompatible(tableSchema, newSchema)) {
+ val tableSchema = HoodieSchemaUtils.createHoodieWriteSchema(schemaUtil.getTableSchema(false).toString, false)
Review Comment:
@the-other-tim-brown Currently the util only takes in a string
<img width="1302" height="484" alt="Screenshot 2025-12-14 at 11 29 15 AM" src="https://github.com/user-attachments/assets/775a2a13-b66f-45da-afba-8bbfe2c9cae6" />
Is the ask then to just make a util that directly operates on the HoodieSchema object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617433717
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -50,7 +51,7 @@ object AvroConversionUtils {
* @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
*/
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
Review Comment:
I thought we were using the equivalent `createGenericRecordToInternalRowConverter` inside the `HoodieSchemaConversionUtils` https://github.com/apache/hudi/pull/17573#discussion_r2612013328? Or is better to remove the hoodischema util method and just modify the `createAvroToInternalRowConverter` instead to take HoodieSchema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649755988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e319753770fd6e626766024383138e3332729da9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208)
* c9377b9fadb42c5ee11099fb01c5cd53f13f8120 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211)
* af3c3c76b7966506eec0e3eb22e565c13a34a770 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3659030712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260)
* 8e82c5dd3f3374e1806f3d358379f9f4e85f174a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628731709
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -49,17 +51,17 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializerInternal(rootType: HoodieSchema,
rootCatalystType: DataType,
positionalFieldMatch: Boolean,
datetimeRebaseSpec: RebaseSpec,
filters: StructFilters) {
- def this(rootAvroType: Schema,
+ def this(rootType: HoodieSchema,
Review Comment:
Eventually we want to get rid of the Avro conversion in Spark so we'll remove these altogether. Thus let them alone would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675627220
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 452d8869cb0deeef79738582648888e57da74584 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
* b9f300a7981e88da2c623aa08fd47e190a6f3374 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632395998
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConversionUtils.scala:
##########
@@ -18,10 +18,15 @@
package org.apache.hudi
Review Comment:
Personally i dont mind changing it but i saw that for other spark class such as `HoodieSparkSqlWriter` the package is `org.apache.hudi`
<img width="899" height="211" alt="Screenshot 2025-12-18 at 11 34 38 AM" src="https://github.com/user-attachments/assets/5b29723c-89da-40e2-8850-d93e81b5f6a2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672091608
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
* 18706cf60f015018b214ea78997b17894198021a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632586381
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroDeserializer.scala:
##########
@@ -17,15 +17,16 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
+class HoodieSpark3_3AvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType)
Review Comment:
keep this as avro schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662041260
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624786447
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
+ // Match existing Avro SchemaConverters behavior: use NULL_VALUE for nullable unions
+ // to avoid serializing "default":null in JSON representation
+ val defaultVal = if (fieldSchema.isNullable) {
+ HoodieJsonProperties.NULL_VALUE
+ } else {
+ null
+ }
+ HoodieSchemaField.of(f.name, fieldSchema, doc, defaultVal)
}
Review Comment:
Have made a change on that line 190 let me know if it makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662836082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302)
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632571681
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -595,7 +674,29 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
public static HoodieSchema projectSchema(HoodieSchema fileSchema, List<String> fields) {
return HoodieSchema.fromAvroSchema(HoodieAvroUtils.projectSchema(fileSchema.toAvroSchema(), fields));
}
+
+ /**
+ * Gets the fully-qualified Avro record name for a Hudi table.
+ * This is equivalent to {@link AvroSchemaUtils#getAvroRecordQualifiedName(String)}
+ * but provides a HoodieSchema-context API.
+ *
+ * <p>The qualified name follows the pattern: hoodie.{tableName}.{tableName}_record
+ * where tableName is sanitized for Avro compatibility.</p>
+ *
+ * @param tableName the Hudi table name
+ * @return the fully-qualified Avro record name (e.g., "hoodie.my_table.my_table_record")
+ * @throws IllegalArgumentException if tableName is null or empty
+ * @since 1.2.0
+ */
+ public static String getRecordQualifiedName(String tableName) {
+ ValidationUtils.checkArgument(tableName != null && !tableName.trim().isEmpty(),
+ "Table name cannot be null or empty");
+
+ // Delegate to AvroSchemaUtils
+ return AvroSchemaUtils.getAvroRecordQualifiedName(tableName);
Review Comment:
@yihua mentioned this will be covered in a follow up task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632518576
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -49,17 +51,17 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializerInternal(rootType: HoodieSchema,
rootCatalystType: DataType,
positionalFieldMatch: Boolean,
datetimeRebaseSpec: RebaseSpec,
filters: StructFilters) {
- def this(rootAvroType: Schema,
+ def this(rootType: HoodieSchema,
Review Comment:
Have made the changes to keep things as they were for these set of classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633316559
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestBootstrap.java:
##########
@@ -62,7 +62,8 @@
import org.apache.hudi.testutils.HoodieMergeOnReadTestUtils;
import org.apache.hudi.testutils.HoodieSparkClientTestBase;
-import org.apache.avro.Schema;
+import org.apache.hudi.common.schema.HoodieSchema;
+
Review Comment:
import grouping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632571681
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -595,7 +674,29 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
public static HoodieSchema projectSchema(HoodieSchema fileSchema, List<String> fields) {
return HoodieSchema.fromAvroSchema(HoodieAvroUtils.projectSchema(fileSchema.toAvroSchema(), fields));
}
+
+ /**
+ * Gets the fully-qualified Avro record name for a Hudi table.
+ * This is equivalent to {@link AvroSchemaUtils#getAvroRecordQualifiedName(String)}
+ * but provides a HoodieSchema-context API.
+ *
+ * <p>The qualified name follows the pattern: hoodie.{tableName}.{tableName}_record
+ * where tableName is sanitized for Avro compatibility.</p>
+ *
+ * @param tableName the Hudi table name
+ * @return the fully-qualified Avro record name (e.g., "hoodie.my_table.my_table_record")
+ * @throws IllegalArgumentException if tableName is null or empty
+ * @since 1.2.0
+ */
+ public static String getRecordQualifiedName(String tableName) {
+ ValidationUtils.checkArgument(tableName != null && !tableName.trim().isEmpty(),
+ "Table name cannot be null or empty");
+
+ // Delegate to AvroSchemaUtils
+ return AvroSchemaUtils.getAvroRecordQualifiedName(tableName);
Review Comment:
@yihua mentioned this will be covered in a follow up task and we dont need to block on this pr
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionHelper.scala:
##########
@@ -41,11 +42,11 @@ class TestAvroConversionHelper extends FunSuite with Matchers {
val dateInputData = Seq(7, 365, 0)
test("Logical type: date") {
- val schema = new Schema.Parser().parse(dateSchema)
- val convertor = AvroConversionUtils.createConverterToRow(schema, AvroConversionUtils.convertAvroSchemaToStructType(schema))
+ val schema = HoodieSchema.parse(dateSchema)
+ val convertor = AvroConversionUtils.createConverterToRow(schema.toAvroSchema, HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(schema))
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632846338
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroDeserializer.scala:
##########
@@ -17,15 +17,16 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
+class HoodieSpark3_3AvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType)
Review Comment:
addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633259161
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -79,6 +79,24 @@ public static HoodieSchema createHoodieWriteSchema(String schema, boolean withOp
return HoodieSchema.fromAvroSchema(avroSchema);
}
+ /**
+ * Creates a write schema for Hudi operations, adding necessary metadata fields.
+ * This is equivalent to HoodieAvroUtils.createHoodieWriteSchema() but operates on HoodieSchema.
+ *
+ * @param schema the base HoodieSchema
+ * @param withOperationField whether to include operation metadata field
+ * @return HoodieSchema configured for write operations with metadata fields added
+ * @throws IllegalArgumentException if schema is null
+ * @since 1.2.0
+ */
+ public static HoodieSchema createHoodieWriteSchema(HoodieSchema schema, boolean withOperationField) {
+ ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
+
+ // Convert to Avro, delegate to existing utility, convert back
+ Schema avroSchema = HoodieAvroUtils.addMetadataFields(schema.toAvroSchema(), withOperationField);
Review Comment:
Let's make sure we add this comment to the follow-up GH task so it is tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675621539
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 452d8869cb0deeef79738582648888e57da74584 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
* b9f300a7981e88da2c623aa08fd47e190a6f3374 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3657816911
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
* 40f3c37722c93a9a37136f128ab5fc135feb34b9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2623545830
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala:
##########
@@ -483,15 +483,15 @@ object ExpressionPayload {
avroDeserializerCache.get()
.get(schema, new Function[Schema, HoodieAvroDeserializer] {
override def apply(t: Schema): HoodieAvroDeserializer =
- sparkAdapter.createAvroDeserializer(schema, convertAvroSchemaToStructType(schema))
+ sparkAdapter.createAvroDeserializer(HoodieSchema.fromAvroSchema(schema), convertHoodieSchemaToDataType(HoodieSchema.fromAvroSchema(schema)))
})
}
private def getAvroSerializerFor(schema: Schema) = {
Review Comment:
Let's make these caches take in `HoodieSchema` as well if possible
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableChangeColumnCommand.scala:
##########
@@ -86,10 +85,10 @@ case class AlterHoodieTableChangeColumnCommand(
Seq.empty[Row]
}
- private def validateSchema(newSchema: Schema, metaClient: HoodieTableMetaClient): Unit = {
+ private def validateSchema(newSchema: HoodieSchema, metaClient: HoodieTableMetaClient): Unit = {
val schemaUtil = new TableSchemaResolver(metaClient)
- val tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchema(false))
- if (!AvroSchemaUtils.isSchemaCompatible(tableSchema, newSchema)) {
+ val tableSchema = HoodieSchemaUtils.createHoodieWriteSchema(schemaUtil.getTableSchema(false).toString, false)
Review Comment:
@rahil-c please address this as well
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala:
##########
@@ -17,7 +17,7 @@
package org.apache.spark.sql.adapter
-import org.apache.hudi.{AvroConversionUtils, DefaultSource, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, Spark3HoodiePartitionCDCFileGroupMapping, Spark3HoodiePartitionFileSliceMapping}
+import org.apache.hudi.{AvroConversionUtils, DefaultSource, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSchemaConversionUtils, Spark3HoodiePartitionCDCFileGroupMapping, Spark3HoodiePartitionFileSliceMapping}
Review Comment:
Is this causing a runtime issue? How did the other PR pass without the import?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2627512344
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -49,17 +51,17 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializerInternal(rootType: HoodieSchema,
Review Comment:
Nitpick: fix indentation to align args
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -17,13 +17,15 @@
package org.apache.spark.sql.avro
Review Comment:
Does this still need to be on the spark path or can we move this to the Hoodie path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628453042
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -58,7 +56,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SparkSession, SQLContext}
import org.apache.spark.sql.HoodieCatalystExpressionUtils.{convertToCatalystExpression, generateUnsafeProjection}
-import org.apache.spark.sql.avro.HoodieSparkAvroSchemaConverters
+import org.apache.spark.sql.avro.{HoodieSparkAvroSchemaConverters, HoodieSparkSchemaConverters}
Review Comment:
Have created subtask here for now under the board https://github.com/apache/hudi/issues/17631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3661281020
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620858866
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -58,7 +56,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SparkSession, SQLContext}
import org.apache.spark.sql.HoodieCatalystExpressionUtils.{convertToCatalystExpression, generateUnsafeProjection}
-import org.apache.spark.sql.avro.HoodieSparkAvroSchemaConverters
+import org.apache.spark.sql.avro.{HoodieSparkAvroSchemaConverters, HoodieSparkSchemaConverters}
Review Comment:
Would it be possible to instead have as a separate removal task similar to https://github.com/apache/hudi/issues/14282? We can have it a fast follow after this that way we can aim to land this large piece first, but let me know what you think.
If you just mean to remove usages of this `HoodieSparkAvroSchemaConverters` in this file alone can do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662688759
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302)
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2623530011
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
@rahil-c let's use the util. Sorry for posing this as a question if that was confusing. The util will allow us to change the behavior consistently across the code in the future if we need to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662464684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302)
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3658063982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40f3c37722c93a9a37136f128ab5fc135feb34b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257)
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620827994
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala:
##########
@@ -94,13 +95,13 @@ trait SparkAdapter extends Serializable {
* Creates instance of [[HoodieAvroSerializer]] providing for ability to serialize
* Spark's [[InternalRow]] into Avro payloads
*/
- def createAvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean): HoodieAvroSerializer
+ def createAvroSerializer(rootCatalystType: DataType, rootHoodieType: HoodieSchema, nullable: Boolean): HoodieAvroSerializer
Review Comment:
I think then `rootType` should suffice, will try this first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620827994
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala:
##########
@@ -94,13 +95,13 @@ trait SparkAdapter extends Serializable {
* Creates instance of [[HoodieAvroSerializer]] providing for ability to serialize
* Spark's [[InternalRow]] into Avro payloads
*/
- def createAvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean): HoodieAvroSerializer
+ def createAvroSerializer(rootCatalystType: DataType, rootHoodieType: HoodieSchema, nullable: Boolean): HoodieAvroSerializer
Review Comment:
I think then `rootType` should suffice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2623647223
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
+ // Match existing Avro SchemaConverters behavior: use NULL_VALUE for nullable unions
+ // to avoid serializing "default":null in JSON representation
+ val defaultVal = if (fieldSchema.isNullable) {
+ HoodieJsonProperties.NULL_VALUE
+ } else {
+ null
+ }
+ HoodieSchemaField.of(f.name, fieldSchema, doc, defaultVal)
}
Review Comment:
On line 190 there is a bug is this code where the comment is added even if the doc string is empty which deviates from the avro path. That is causing some of the failures in CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624204110
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
+ // Match existing Avro SchemaConverters behavior: use NULL_VALUE for nullable unions
+ // to avoid serializing "default":null in JSON representation
+ val defaultVal = if (fieldSchema.isNullable) {
+ HoodieJsonProperties.NULL_VALUE
+ } else {
+ null
+ }
+ HoodieSchemaField.of(f.name, fieldSchema, doc, defaultVal)
}
Review Comment:
thanks for catching this, will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644502386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c5682b88e5c591d07223826f552502d85517001 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147)
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644096062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c5682b88e5c591d07223826f552502d85517001 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632328234
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -369,15 +367,15 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
val fileSplits = collectFileSplits(partitionFilters, dataFilters)
- val tableAvroSchemaStr = tableAvroSchema.toString
+ val tableSchemaStr = tableSchema.toString
- val tableSchema = HoodieTableSchema(tableStructSchema, tableAvroSchemaStr, internalSchemaOpt)
+ val schema = HoodieTableSchema(tableStructSchema, tableSchemaStr, internalSchemaOpt)
Review Comment:
Have made the change to store HoodieSchema inside `HoodieTableSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633549846
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroSerializer.scala:
##########
@@ -17,13 +17,14 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean)
+class HoodieSpark3_3AvroSerializer(rootCatalystType: DataType, rootType: HoodieSchema, nullable: Boolean)
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632392884
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -307,6 +325,31 @@ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema s
return HoodieSchemaField.fromAvroField(avroField);
}
+ /**
+ * Creates a new schema field with the specified properties, including field order.
+ * This is equivalent to HoodieAvroUtils.createNewSchemaField() but returns HoodieSchemaField.
+ *
+ * @param name field name
+ * @param schema field schema
+ * @param doc field documentation (can be null)
+ * @param defaultValue default value (can be null)
+ * @param order field order for sorting
+ * @return new HoodieSchemaField instance
+ * @throws IllegalArgumentException if name, schema, or order is null/empty
+ * @since 1.2.0
+ */
+ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema schema,
+ String doc, Object defaultValue, HoodieFieldOrder order) {
+ ValidationUtils.checkArgument(name != null && !name.isEmpty(), "Field name cannot be null or empty");
+ ValidationUtils.checkArgument(schema != null, "Field schema cannot be null");
+ ValidationUtils.checkArgument(order != null, "Field order cannot be null");
+
+ // Delegate to HoodieAvroUtils
+ Schema.Field avroField = HoodieAvroUtils.createNewSchemaField(
+ name, schema.toAvroSchema(), doc, defaultValue, order.toAvroOrder());
+ return HoodieSchemaField.fromAvroField(avroField);
+ }
Review Comment:
@yihua aligned with ethan here we can keep this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672112426
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
* 18706cf60f015018b214ea78997b17894198021a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632556458
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConversionUtils.scala:
##########
@@ -18,10 +18,15 @@
package org.apache.hudi
Review Comment:
@yihua requested a follow up github task for clean up of certain spark classes to be under a more suitable prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3663398394
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628733394
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -23,12 +23,12 @@
import org.apache.hudi.avro.model.HoodieClusteringPlan;
import org.apache.hudi.client.WriteStatus;
import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.schema.HoodieSchema;
Review Comment:
This import is out of order.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestWriteClient.java:
##########
@@ -31,7 +31,8 @@
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.testutils.HoodieSparkClientTestBase;
-import org.apache.avro.Schema;
+import org.apache.hudi.common.schema.HoodieSchema;
Review Comment:
nit: group import with previous group
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649772802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9377b9fadb42c5ee11099fb01c5cd53f13f8120 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211)
* af3c3c76b7966506eec0e3eb22e565c13a34a770 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617737166
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -582,6 +618,7 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
Review Comment:
Maybe there is some other issue with the javadoc then? It does not make sense to add a newline between the javadoc and the method. We don't see that elsewhere in the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3656385569
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 790babc0c83f242c3265bdbb5866e5b71230fb25 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229)
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620313384
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -50,7 +51,7 @@ object AvroConversionUtils {
* @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
*/
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
Review Comment:
Ok, I missed that this was just remaining to ease the migration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643589289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 204f6219585adb778e51680eaf3bcedeb8e8d178 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649706258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188)
* e319753770fd6e626766024383138e3332729da9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649814739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af3c3c76b7966506eec0e3eb22e565c13a34a770 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2627497838
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
When we're just copying the value, we can use the `createNewSchemaField` that takes in the field directly so it is less verbose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631515348
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -307,6 +325,31 @@ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema s
return HoodieSchemaField.fromAvroField(avroField);
}
+ /**
+ * Creates a new schema field with the specified properties, including field order.
+ * This is equivalent to HoodieAvroUtils.createNewSchemaField() but returns HoodieSchemaField.
+ *
+ * @param name field name
+ * @param schema field schema
+ * @param doc field documentation (can be null)
+ * @param defaultValue default value (can be null)
+ * @param order field order for sorting
+ * @return new HoodieSchemaField instance
+ * @throws IllegalArgumentException if name, schema, or order is null/empty
+ * @since 1.2.0
+ */
+ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema schema,
+ String doc, Object defaultValue, HoodieFieldOrder order) {
+ ValidationUtils.checkArgument(name != null && !name.isEmpty(), "Field name cannot be null or empty");
+ ValidationUtils.checkArgument(schema != null, "Field schema cannot be null");
+ ValidationUtils.checkArgument(order != null, "Field order cannot be null");
+
+ // Delegate to HoodieAvroUtils
+ Schema.Field avroField = HoodieAvroUtils.createNewSchemaField(
+ name, schema.toAvroSchema(), doc, defaultValue, order.toAvroOrder());
+ return HoodieSchemaField.fromAvroField(avroField);
+ }
Review Comment:
I think the only logic difference between the two methods is that one is passing the order field and the other is not.
<img width="975" height="777" alt="Screenshot 2025-12-18 at 7 03 14 AM" src="https://github.com/user-attachments/assets/99236898-8364-4885-9fa4-4242fca398c0" />, so im not sure if its a question of logic resuse. I think we should keep this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3651615691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a49e78d78fb9a210b6405236482916de1bc1a066 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218)
* e6a6deb8869158c0d0e18fc8e48d05009656c952 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633260231
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -307,6 +325,31 @@ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema s
return HoodieSchemaField.fromAvroField(avroField);
}
+ /**
+ * Creates a new schema field with the specified properties, including field order.
+ * This is equivalent to HoodieAvroUtils.createNewSchemaField() but returns HoodieSchemaField.
+ *
+ * @param name field name
+ * @param schema field schema
+ * @param doc field documentation (can be null)
+ * @param defaultValue default value (can be null)
+ * @param order field order for sorting
+ * @return new HoodieSchemaField instance
+ * @throws IllegalArgumentException if name, schema, or order is null/empty
+ * @since 1.2.0
+ */
+ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema schema,
+ String doc, Object defaultValue, HoodieFieldOrder order) {
+ ValidationUtils.checkArgument(name != null && !name.isEmpty(), "Field name cannot be null or empty");
+ ValidationUtils.checkArgument(schema != null, "Field schema cannot be null");
+ ValidationUtils.checkArgument(order != null, "Field order cannot be null");
+
+ // Delegate to HoodieAvroUtils
+ Schema.Field avroField = HoodieAvroUtils.createNewSchemaField(
+ name, schema.toAvroSchema(), doc, defaultValue, order.toAvroOrder());
+ return HoodieSchemaField.fromAvroField(avroField);
+ }
Review Comment:
Given that this is the delegation, it is OK for now.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -193,8 +192,8 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
.withBaseFileOption(baseFileOption)
.withPartitionPath(partitionPath)
.withProps(properties)
- .withDataSchema(HoodieSchema.parse(tableSchema.avroSchemaStr))
- .withRequestedSchema(HoodieSchema.parse(requiredSchema.avroSchemaStr))
+ .withDataSchema(tableSchema.tableSchema)
+ .withRequestedSchema(requiredSchema.tableSchema)
Review Comment:
Should this be renamed from `tableSchema` to `schema` since it's not always table schema?
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -582,6 +618,7 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
Review Comment:
@rahil-c Could you check this separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633304473
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/TestHoodieMergeHandleWithSparkMerger.java:
##########
@@ -171,19 +171,19 @@ public HoodieWriteConfig getWriteConfig(Schema avroSchema, String recordMergerIm
return getConfigBuilder(true)
.withPath(basePath())
- .withSchema(avroSchema.toString())
+ .withSchema(schema.toString())
.withProps(properties)
.build();
}
- public HoodieWriteConfig buildDefaultWriteConfig(Schema avroSchema) {
- HoodieWriteConfig writeConfig = getWriteConfig(avroSchema, DefaultMerger.class.getName(), HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID, RecordMergeMode.EVENT_TIME_ORDERING);
+ public HoodieWriteConfig buildDefaultWriteConfig(HoodieSchema hoodieSchema) {
+ HoodieWriteConfig writeConfig = getWriteConfig(hoodieSchema, DefaultMerger.class.getName(), HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID, RecordMergeMode.EVENT_TIME_ORDERING);
metaClient = getHoodieMetaClient(storageConf(), basePath(), HoodieTableType.MERGE_ON_READ, writeConfig.getProps());
return writeConfig;
}
- public HoodieWriteConfig buildCustomWriteConfig(Schema avroSchema) {
- HoodieWriteConfig writeConfig = getWriteConfig(avroSchema, CustomMerger.class.getName(), HoodieRecordMerger.CUSTOM_MERGE_STRATEGY_UUID, RecordMergeMode.CUSTOM);
+ public HoodieWriteConfig buildCustomWriteConfig(HoodieSchema hoodieSchema) {
+ HoodieWriteConfig writeConfig = getWriteConfig(hoodieSchema, CustomMerger.class.getName(), HoodieRecordMerger.CUSTOM_MERGE_STRATEGY_UUID, RecordMergeMode.CUSTOM);
Review Comment:
nit: `hoodieSchema` -> `schema`
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -212,8 +211,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
- val requestedAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(requestedSchema, sanitizedTableName), exclusionFields)
- val dataAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(dataSchema, sanitizedTableName), exclusionFields)
+ val requestedHoodieSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(requestedSchema, sanitizedTableName), exclusionFields)
+ val dataHoodieSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(dataSchema, sanitizedTableName), exclusionFields)
Review Comment:
```suggestion
val requestedSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(requestedSchema, sanitizedTableName), exclusionFields)
val dataSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(dataSchema, sanitizedTableName), exclusionFields)
```
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -51,12 +51,12 @@ object SparkHelpers {
val sourceRecords = HoodieIOFactory.getIOFactory(storage)
.getFileFormatUtils(HoodieFileFormat.PARQUET)
.readAvroRecords(storage, sourceFile).asScala
- val schema: Schema = sourceRecords.head.getSchema
+ val schema: HoodieSchema = HoodieSchema.fromAvroSchema(sourceRecords.head.getSchema)
val filter: BloomFilter = BloomFilterFactory.createBloomFilter(
BLOOM_FILTER_NUM_ENTRIES_VALUE.defaultValue.toInt, BLOOM_FILTER_FPP_VALUE.defaultValue.toDouble,
BLOOM_FILTER_DYNAMIC_MAX_ENTRIES.defaultValue.toInt, BLOOM_FILTER_TYPE.defaultValue);
- val writeSupport: HoodieAvroWriteSupport[_] = new HoodieAvroWriteSupport(getAvroSchemaConverter(conf.unwrap()).convert(schema),
- schema, Option.of(filter), new Properties())
+ val writeSupport: HoodieAvroWriteSupport[_] = new HoodieAvroWriteSupport(getAvroSchemaConverter(conf.unwrap()).convert(schema.getAvroSchema),
+ schema.getAvroSchema, Option.of(filter), new Properties())
Review Comment:
Seems no need to change this as it merely converting the the Avro schema to `HoodieSchema` and then back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675632383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
}, {
"hash" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 452d8869cb0deeef79738582648888e57da74584 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
* b9f300a7981e88da2c623aa08fd47e190a6f3374 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373)
* a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672489945
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18706cf60f015018b214ea78997b17894198021a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356)
* e66b1f84650709425024b101fe319b5c547d3746 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672514461
@the-other-tim-brown @yihua I am hoping we can avoid any other additional scope or changes for this PR so we land this and unblock other streams. All comments ideally should be addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620439344
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableChangeColumnCommand.scala:
##########
@@ -86,10 +85,10 @@ case class AlterHoodieTableChangeColumnCommand(
Seq.empty[Row]
}
- private def validateSchema(newSchema: Schema, metaClient: HoodieTableMetaClient): Unit = {
+ private def validateSchema(newSchema: HoodieSchema, metaClient: HoodieTableMetaClient): Unit = {
val schemaUtil = new TableSchemaResolver(metaClient)
- val tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchema(false))
- if (!AvroSchemaUtils.isSchemaCompatible(tableSchema, newSchema)) {
+ val tableSchema = HoodieSchemaUtils.createHoodieWriteSchema(schemaUtil.getTableSchema(false).toString, false)
Review Comment:
Yes, we have been creating utils that match existing utils so we can do that here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3658215777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620877683
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.datasources.parquet
-import org.apache.hudi.{AvroConversionUtils, HoodieFileIndex, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSparkUtils, HoodieTableSchema, SparkAdapterSupport, SparkFileFormatInternalRowReaderContext}
+import org.apache.hudi.{AvroConversionUtils, HoodieFileIndex, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSchemaConversionUtils, HoodieSparkUtils, HoodieTableSchema, SparkAdapterSupport, SparkFileFormatInternalRowReaderContext}
Review Comment:
I think the ask makes sense, just want to mention that it seems to be used in large amount of places in src and test classes
<img width="1135" height="974" alt="Screenshot 2025-12-15 at 1 15 32 PM" src="https://github.com/user-attachments/assets/22d1673c-d401-4e64-8104-6c94c7a4b498" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620858866
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -58,7 +56,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SparkSession, SQLContext}
import org.apache.spark.sql.HoodieCatalystExpressionUtils.{convertToCatalystExpression, generateUnsafeProjection}
-import org.apache.spark.sql.avro.HoodieSparkAvroSchemaConverters
+import org.apache.spark.sql.avro.{HoodieSparkAvroSchemaConverters, HoodieSparkSchemaConverters}
Review Comment:
Would it be possible to instead have as a separate removal task similar to https://github.com/apache/hudi/issues/14282?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662469755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302)
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3647965503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b766aed2cb16e04408b1fba1a83f8673177836be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184)
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2616084557
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:
##########
@@ -49,13 +49,13 @@ import scala.collection.JavaConverters._
*
* PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
*/
-private[sql] class AvroDeserializer(rootAvroType: Schema,
+private[sql] class AvroDeserializer(rootAvroType: HoodieSchema,
Review Comment:
As discussed offline, given that the `AvroSerializer`, `AvroDeserializer`, and other relevant classes are used to do conversion between `GenericRecord` / `IndexedRecord` and Spark Rows, it is better to keep these classes intact by taking Avro schema as they are needed for Avro serde, and there is little value using `HoodieSchema` inside these classes. The callers can take `HoodieSchema` and pass in the converted Avro schema to these classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632035692
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -369,15 +367,15 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
val fileSplits = collectFileSplits(partitionFilters, dataFilters)
- val tableAvroSchemaStr = tableAvroSchema.toString
+ val tableSchemaStr = tableSchema.toString
- val tableSchema = HoodieTableSchema(tableStructSchema, tableAvroSchemaStr, internalSchemaOpt)
+ val schema = HoodieTableSchema(tableStructSchema, tableSchemaStr, internalSchemaOpt)
Review Comment:
I can look into storing the HoodieSchema directly in `HoodieTableSchema`, just need to confirm how the string is utilized and how many files it touches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675122090
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367)
* 452d8869cb0deeef79738582648888e57da74584 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675868465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
}, {
"hash" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374",
"triggerID" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 452d8869cb0deeef79738582648888e57da74584 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
* b9f300a7981e88da2c623aa08fd47e190a6f3374 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373)
* a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675948067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
}, {
"hash" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374",
"triggerID" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9f300a7981e88da2c623aa08fd47e190a6f3374 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373)
* a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675139762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 452d8869cb0deeef79738582648888e57da74584 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631339320
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
Review Comment:
<img width="763" height="188" alt="Screenshot 2025-12-18 at 6 20 30 AM" src="https://github.com/user-attachments/assets/decefdb9-a69d-4807-a01f-06c396739f66" />
There are no production usages of it, so it can be removed. I think when looking at HoodieAvroUtils we also did not have any three arg constructor like this so I am thinking it might be less confusing and have the caller be explicit on what they want default val as.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3663119381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24dda257806ad8dae89de72faf95f04e02242399 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304)
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3665683574
@the-other-tim-brown @yihua can you guys take a look today when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3663058226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24dda257806ad8dae89de72faf95f04e02242399 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304)
* ed494441741a5aee888a3a384acbee9d033a19ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2627876904
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializerInternal.scala:
##########
@@ -17,13 +17,15 @@
package org.apache.spark.sql.avro
Review Comment:
I believe this will unfortunately have to still be on the spark path as I see in this class usage of certain spark types such as `TimeStampNTZ` which is private in spark 3.3. https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L152
Its basically doing the same trick i believe as what we saw in the spark schema converter pr: https://github.com/apache/hudi/pull/17475#discussion_r2606937790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631877261
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSourceDefaults.scala:
##########
@@ -625,14 +624,14 @@ class TestDataSourceDefaults extends ScalaAssertionSupport {
val props = new TypedProperties()
OrderingFieldsTestUtils.setOrderingFieldsConfig(props, key, "favoriteIntNumber")
val baseOrderingVal: Object = baseRecord.get("favoriteIntNumber")
- val fieldSchema: Schema = baseRecord.getSchema().getField("favoriteIntNumber").schema()
+ val fieldSchema: HoodieSchema = HoodieSchema.fromAvroSchema(baseRecord.getSchema().getField("favoriteIntNumber").schema())
- val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
+ val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
Review Comment:
Without the prefix currently there is another `HoodieSchemaUtils` within the hudi spark module which compiler gets confused(see screenshot). https://github.com/apache/hudi/blame/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSchemaUtils.scala
<img width="731" height="278" alt="Screenshot 2025-12-18 at 7 49 25 AM" src="https://github.com/user-attachments/assets/b6d01093-57df-4211-83c3-e67a758e008d" />
i think we need to either rename the spark's HoodieSchemaUtils class to something else or use full qualified path to the new util we have in common.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSourceDefaults.scala:
##########
@@ -625,14 +624,14 @@ class TestDataSourceDefaults extends ScalaAssertionSupport {
val props = new TypedProperties()
OrderingFieldsTestUtils.setOrderingFieldsConfig(props, key, "favoriteIntNumber")
val baseOrderingVal: Object = baseRecord.get("favoriteIntNumber")
- val fieldSchema: Schema = baseRecord.getSchema().getField("favoriteIntNumber").schema()
+ val fieldSchema: HoodieSchema = HoodieSchema.fromAvroSchema(baseRecord.getSchema().getField("favoriteIntNumber").schema())
- val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
+ val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
val laterRecord = SchemaTestUtil
.generateAvroRecordFromJson(schema, 2, "001", "f1")
val laterOrderingVal: Object = laterRecord.get("favoriteIntNumber")
- val newerPayload = new OverwriteWithLatestAvroPayload(laterRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, laterOrderingVal, false).asInstanceOf[Comparable[_]])
+ val newerPayload = new OverwriteWithLatestAvroPayload(laterRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, laterOrderingVal, false).asInstanceOf[Comparable[_]])
Review Comment:
see comment above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3671817633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
* 18706cf60f015018b214ea78997b17894198021a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631908850
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
+ val structFromData = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(org.apache.hudi.common.schema.HoodieSchemaUtils.removeMetadataFields(schemaFromData))
Review Comment:
see comment https://github.com/apache/hudi/pull/17573#discussion_r2631877261
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643519944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 204f6219585adb778e51680eaf3bcedeb8e8d178 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3652380664
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 790babc0c83f242c3265bdbb5866e5b71230fb25 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3651771109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6a6deb8869158c0d0e18fc8e48d05009656c952 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3647753644
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
* b766aed2cb16e04408b1fba1a83f8673177836be UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649720063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e319753770fd6e626766024383138e3332729da9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2612034841
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableChangeColumnCommand.scala:
##########
@@ -86,10 +85,10 @@ case class AlterHoodieTableChangeColumnCommand(
Seq.empty[Row]
}
- private def validateSchema(newSchema: Schema, metaClient: HoodieTableMetaClient): Unit = {
+ private def validateSchema(newSchema: HoodieSchema, metaClient: HoodieTableMetaClient): Unit = {
val schemaUtil = new TableSchemaResolver(metaClient)
- val tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchema(false))
- if (!AvroSchemaUtils.isSchemaCompatible(tableSchema, newSchema)) {
+ val tableSchema = HoodieSchemaUtils.createHoodieWriteSchema(schemaUtil.getTableSchema(false).toString, false)
Review Comment:
Can we avoid converting the schema to string here?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,8 +211,9 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
- val requestedAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(requestedSchema, sanitizedTableName), exclusionFields)
- val dataAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(dataSchema, sanitizedTableName), exclusionFields)
+ //TODO add util for this in HoodieSchema
+ val requestedAvroSchema = AvroSchemaUtils.pruneDataSchema(hoodieTableSchema.getAvroSchema, AvroConversionUtils.convertStructTypeToAvroSchema(requestedSchema, sanitizedTableName), exclusionFields)
Review Comment:
HoodieSchemaUtils now has a `pruneDataSchema`
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/HDFSParquetImporterUtils.java:
##########
@@ -183,7 +183,7 @@ public JavaRDD<HoodieRecord<HoodieRecordPayload>> buildHoodieRecordsForImport(Ja
job.getConfiguration().set(FileInputFormat.INPUT_DIR_RECURSIVE, "true");
// To parallelize reading file status.
job.getConfiguration().set(FileInputFormat.LIST_STATUS_NUM_THREADS, "1024");
- AvroReadSupport.setAvroReadSchema(jsc.hadoopConfiguration(), (new Schema.Parser().parse(schemaStr)));
+ AvroReadSupport.setAvroReadSchema(jsc.hadoopConfiguration(), (HoodieSchema.parse(schemaStr).getAvroSchema()));
Review Comment:
nitpick: outer `()` is unnecessary
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConversionUtils.scala:
##########
@@ -194,4 +198,19 @@ object HoodieSchemaConversionUtils {
case _ => dataType
}
}
+
+ /**
+ * Creates a converter from GenericRecord to InternalRow using HoodieSchema.
+ * This is equivalent to AvroConversionUtils.createAvroToInternalRowConverter() but accepts HoodieSchema.
+ *
+ * @param requiredAvroSchema the HoodieSchema to use for deserialization
+ * @param requiredRowSchema the Spark StructType for the output InternalRow
+ * @return a function that converts GenericRecord to Option[InternalRow]
+ */
+ def createHoodieSchemaToInternalRowConverter(requiredAvroSchema: HoodieSchema, requiredRowSchema: StructType): GenericRecord => Option[InternalRow] = {
Review Comment:
I think naming this something like `createGenericRecordToInternalRowConverter` is more fitting. We are not converting from the HoodieSchema to a row.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
I think we'll still need the createNewField logic since the schema is backed by Avro
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
+ fieldMap(col)
}.toList
- val requiredAvroSchema = Schema.createRecord(avroSchema.getName, avroSchema.getDoc,
- avroSchema.getNamespace, avroSchema.isError, requiredFields.asJava)
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
- (requiredAvroSchema, requiredStructSchema, InternalSchema.getEmptyInternalSchema)
+ val fieldsJava = new java.util.ArrayList[HoodieSchemaField]()
Review Comment:
Can we just use `asJava`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643693047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 204f6219585adb778e51680eaf3bcedeb8e8d178 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143)
* 8b3fff68c0346d4464e75e29b62578dca730043a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644536806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2616542014
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -573,6 +573,42 @@ public static HoodieSchema getRecordKeySchema() {
return RECORD_KEY_SCHEMA;
}
+ /**
+ * Converts field values for specific data types with logical type handling.
+ * This is equivalent to HoodieAvroUtils.convertValueForSpecificDataTypes() but operates on HoodieSchema.
+ * <p>
+ * Handles special conversions for Avro logical types:
+ * <ul>
+ * <li>Date type - converts epoch day integer to LocalDate</li>
+ * <li>Timestamp types - converts epoch milliseconds/microseconds to Timestamp</li>
+ * <li>Decimal type - converts bytes/fixed to BigDecimal</li>
+ * </ul>
+ *
+ * @param fieldSchema the field schema
+ * @param fieldValue the field value to convert
+ * @param consistentLogicalTimestampEnabled whether to use consistent logical timestamp handling
+ * @return converted value for logical types, or original value
+ * @throws IllegalStateException if fieldValue is null but schema is not nullable
+ * @since 1.2.0
+ */
+ public static Object convertValueForSpecificDataTypes(HoodieSchema fieldSchema,
Review Comment:
Make sure there are unit tests for this
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -582,6 +618,7 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
Review Comment:
remove this extra newline
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConversionUtils.scala:
##########
@@ -194,4 +198,19 @@ object HoodieSchemaConversionUtils {
case _ => dataType
}
}
+
+ /**
+ * Creates a converter from GenericRecord to InternalRow using HoodieSchema.
+ * This is equivalent to AvroConversionUtils.createAvroToInternalRowConverter() but accepts HoodieSchema.
+ *
+ * @param requiredAvroSchema the HoodieSchema to use for deserialization
Review Comment:
Update the naming of this?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -50,7 +51,7 @@ object AvroConversionUtils {
* @return converter accepting Avro payload and transforming it into a Catalyst one (in the form of [[InternalRow]])
*/
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
Review Comment:
Can we have the methods in this class take in a HoodieSchema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649752218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e319753770fd6e626766024383138e3332729da9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208)
* c9377b9fadb42c5ee11099fb01c5cd53f13f8120 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617417565
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
yea i think i faced an issue in some of the tests so added this https://github.com/apache/hudi/commit/9fe07c055ee605e6d981918e686bb03f6de4070c
```
// We have to create a new HoodieSchemaField since Avro schemas can't share field
// instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
HoodieSchemaField.of(f.name(), f.schema(), f.doc().orElse(null), f.defaultVal().orElse(null))
}.toList
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617533353
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -582,6 +618,7 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
Review Comment:
it seems this caused some check style issue for me when i removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624202988
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala:
##########
@@ -17,7 +17,7 @@
package org.apache.spark.sql.adapter
-import org.apache.hudi.{AvroConversionUtils, DefaultSource, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, Spark3HoodiePartitionCDCFileGroupMapping, Spark3HoodiePartitionFileSliceMapping}
+import org.apache.hudi.{AvroConversionUtils, DefaultSource, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSchemaConversionUtils, Spark3HoodiePartitionCDCFileGroupMapping, Spark3HoodiePartitionFileSliceMapping}
Review Comment:
<img width="1307" height="392" alt="Screenshot 2025-12-16 at 9 31 12 AM" src="https://github.com/user-attachments/assets/a21f3c93-d045-40b6-bdb7-3718782e399e" />
Actually not sure why this was added, since not being used. Will remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632846859
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620514215
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -428,7 +427,7 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
val orderedUpdatedFieldSeq = getOrderedUpdatedFields(updatedFieldSet)
writeParams ++= Seq(
WRITE_PARTIAL_UPDATE_SCHEMA.key ->
- HoodieAvroUtils.generateProjectionSchema(fullSchema, orderedUpdatedFieldSeq.asJava).toString
+ HoodieAvroUtils.generateProjectionSchema(fullSchema.getAvroSchema, orderedUpdatedFieldSeq.asJava).toString
Review Comment:
Let's use the `HoodieSchemaUtils` here for the `generateProjectionSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662961897
@the-other-tim-brown @yihua For the integration test was failing for
```
Error: testAvroFileSinkWriter Time elapsed: 3.125 s <<< ERROR!
org.apache.spark.SparkException:
Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (runnervm6qbrg.zfkur4ax2nlu5onhaw0pux0jqe.ex.internal.cloudapp.net executor driver): java.lang.NoSuchMethodError: org.apache.spark.sql.avro.AvroSerializer.<init>(Lorg/apache/spark/sql/types/DataType;Lorg/apache/hudi/common/schema/HoodieSchema;Z)V
at org.apache.spark.sql.avro.HoodieSpark3_5AvroSerializer.<init>(HoodieSpark3_5AvroSerializer.scala:28)
at org.apache.spark.sql.adapter.Spark3_5Adapter.createAvroSerializer(Spark3_5Adapter.scala:92)
at org.apache.hudi.AvroConversionUtils$.$anonfun$createInternalRowToAvroConverter$1(AvroConversionUtils.scala:70)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.hudi.AvroConversionUtils$.createInternalRowToAvroConverter(AvroConversionUtils.scala:82)
at org.apache.hudi.HoodieSparkUtils$.$anonfun$createRdd$2(HoodieSparkUtils.scala:119)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:858)
```
I think this is due to the fact that Hudi has its own class called `AvroSerializer` which has the same naming as the spark avro `AvroSerializer` class and likely runtime was getting confused by this and loading the wrong class which was not using the constructor with HoodieSchema.
<img width="2038" height="476" alt="Screenshot 2025-12-16 at 3 57 45 PM" src="https://github.com/user-attachments/assets/78126af3-2ff9-427b-9630-5bd9aafddf93" />
Im gonna push a change to see if the rename the class works since it worked locally
<img width="2433" height="478" alt="Screenshot 2025-12-16 at 3 59 22 PM" src="https://github.com/user-attachments/assets/642b52f0-6cca-45be-a9c8-b96461446f25" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3657936433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
* 40f3c37722c93a9a37136f128ab5fc135feb34b9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257)
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3659026791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5500fd7a2d0676063d950f5e51cee10ebbe1e0f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260)
* 8e82c5dd3f3374e1806f3d358379f9f4e85f174a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662047192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624411003
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala:
##########
@@ -483,15 +483,15 @@ object ExpressionPayload {
avroDeserializerCache.get()
.get(schema, new Function[Schema, HoodieAvroDeserializer] {
override def apply(t: Schema): HoodieAvroDeserializer =
- sparkAdapter.createAvroDeserializer(schema, convertAvroSchemaToStructType(schema))
+ sparkAdapter.createAvroDeserializer(HoodieSchema.fromAvroSchema(schema), convertHoodieSchemaToDataType(HoodieSchema.fromAvroSchema(schema)))
})
}
private def getAvroSerializerFor(schema: Schema) = {
Review Comment:
Will try to make this change for this class only to have caches take in HoodieSchema. However i think my concern is this might cascade and touch other classes potentially(will give first attempt).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2635098331
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/ColumnStatIndexTestBase.scala:
##########
@@ -36,7 +36,7 @@ import org.apache.hudi.storage.StoragePath
import org.apache.hudi.storage.hadoop.HadoopStorageConfiguration
import org.apache.hudi.testutils.{HoodieSparkClientTestBase, LogFileColStatsTestUtil}
-import org.apache.avro.Schema
+import org.apache.avro.{Schema}
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_5AvroSerializer.scala:
##########
@@ -17,6 +17,8 @@
package org.apache.spark.sql.avro
+import org.apache.hudi.common.schema.HoodieSchema
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCDataFrameSuite.scala:
##########
@@ -21,6 +21,7 @@ package org.apache.hudi.functional.cdc
import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions.{MOR_TABLE_TYPE_OPT_VAL, PARTITIONPATH_FIELD_OPT_KEY, PRECOMBINE_FIELD_OPT_KEY, RECORDKEY_FIELD_OPT_KEY}
import org.apache.hudi.QuickstartUtils.getQuickstartWriteConfigs
+import org.apache.hudi.common.schema.HoodieSchema
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestBootstrap.java:
##########
@@ -62,7 +62,8 @@
import org.apache.hudi.testutils.HoodieMergeOnReadTestUtils;
import org.apache.hudi.testutils.HoodieSparkClientTestBase;
-import org.apache.avro.Schema;
+import org.apache.hudi.common.schema.HoodieSchema;
+
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/TestHoodieMergeHandleWithSparkMerger.java:
##########
@@ -171,19 +171,19 @@ public HoodieWriteConfig getWriteConfig(Schema avroSchema, String recordMergerIm
return getConfigBuilder(true)
.withPath(basePath())
- .withSchema(avroSchema.toString())
+ .withSchema(schema.toString())
.withProps(properties)
.build();
}
- public HoodieWriteConfig buildDefaultWriteConfig(Schema avroSchema) {
- HoodieWriteConfig writeConfig = getWriteConfig(avroSchema, DefaultMerger.class.getName(), HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID, RecordMergeMode.EVENT_TIME_ORDERING);
+ public HoodieWriteConfig buildDefaultWriteConfig(HoodieSchema hoodieSchema) {
+ HoodieWriteConfig writeConfig = getWriteConfig(hoodieSchema, DefaultMerger.class.getName(), HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID, RecordMergeMode.EVENT_TIME_ORDERING);
metaClient = getHoodieMetaClient(storageConf(), basePath(), HoodieTableType.MERGE_ON_READ, writeConfig.getProps());
return writeConfig;
}
- public HoodieWriteConfig buildCustomWriteConfig(Schema avroSchema) {
- HoodieWriteConfig writeConfig = getWriteConfig(avroSchema, CustomMerger.class.getName(), HoodieRecordMerger.CUSTOM_MERGE_STRATEGY_UUID, RecordMergeMode.CUSTOM);
+ public HoodieWriteConfig buildCustomWriteConfig(HoodieSchema hoodieSchema) {
+ HoodieWriteConfig writeConfig = getWriteConfig(hoodieSchema, CustomMerger.class.getName(), HoodieRecordMerger.CUSTOM_MERGE_STRATEGY_UUID, RecordMergeMode.CUSTOM);
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/SparkHelpers.scala:
##########
@@ -51,12 +51,12 @@ object SparkHelpers {
val sourceRecords = HoodieIOFactory.getIOFactory(storage)
.getFileFormatUtils(HoodieFileFormat.PARQUET)
.readAvroRecords(storage, sourceFile).asScala
- val schema: Schema = sourceRecords.head.getSchema
+ val schema: HoodieSchema = HoodieSchema.fromAvroSchema(sourceRecords.head.getSchema)
val filter: BloomFilter = BloomFilterFactory.createBloomFilter(
BLOOM_FILTER_NUM_ENTRIES_VALUE.defaultValue.toInt, BLOOM_FILTER_FPP_VALUE.defaultValue.toDouble,
BLOOM_FILTER_DYNAMIC_MAX_ENTRIES.defaultValue.toInt, BLOOM_FILTER_TYPE.defaultValue);
- val writeSupport: HoodieAvroWriteSupport[_] = new HoodieAvroWriteSupport(getAvroSchemaConverter(conf.unwrap()).convert(schema),
- schema, Option.of(filter), new Properties())
+ val writeSupport: HoodieAvroWriteSupport[_] = new HoodieAvroWriteSupport(getAvroSchemaConverter(conf.unwrap()).convert(schema.getAvroSchema),
+ schema.getAvroSchema, Option.of(filter), new Properties())
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2635537485
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,9 +210,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val exclusionFields = new java.util.HashSet[String]()
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
- val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
Review Comment:
Changes made:
`dataSchema` (in method signature) -> `dataStructType`
`requestedSchema` -> `requestedStructType`
These changes make sense to me as the type is now described in the variable name and is more descriptive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3673642948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631362482
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -79,6 +79,24 @@ public static HoodieSchema createHoodieWriteSchema(String schema, boolean withOp
return HoodieSchema.fromAvroSchema(avroSchema);
}
+ /**
+ * Creates a write schema for Hudi operations, adding necessary metadata fields.
+ * This is equivalent to HoodieAvroUtils.createHoodieWriteSchema() but operates on HoodieSchema.
+ *
+ * @param schema the base HoodieSchema
+ * @param withOperationField whether to include operation metadata field
+ * @return HoodieSchema configured for write operations with metadata fields added
+ * @throws IllegalArgumentException if schema is null
+ * @since 1.2.0
+ */
+ public static HoodieSchema createHoodieWriteSchema(HoodieSchema schema, boolean withOperationField) {
+ ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
+
+ // Convert to Avro, delegate to existing utility, convert back
+ Schema avroSchema = HoodieAvroUtils.addMetadataFields(schema.toAvroSchema(), withOperationField);
Review Comment:
I can remove this method and use `HoodieSchemaUtils#addMetadataFields`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3671592357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
* e7b1643815be3c3dac004de710bb9343658f9d66 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3673415346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e66b1f84650709425024b101fe319b5c547d3746 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359)
* c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633546757
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroDeserializer.scala:
##########
@@ -17,15 +17,16 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
+class HoodieSpark3_3AvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType)
Review Comment:
will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675126318
> I've addressed the comments, but i do not have permissions to push to @rahil-c's branch.
>
> If anyone has permissions, you can apply the changes using the patch below: [Address_styling_comments_1.patch](https://github.com/user-attachments/files/24258742/Address_styling_comments_1.patch)
granted permissions thanks again @voonhous
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643534478
@the-other-tim-brown the pr is still in draft state but hopefully 90 percent complete so you can see if directionally things make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3643754293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b3fff68c0346d4464e75e29b62578dca730043a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644051852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b3fff68c0346d4464e75e29b62578dca730043a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146)
* 4c5682b88e5c591d07223826f552502d85517001 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644058126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b3fff68c0346d4464e75e29b62578dca730043a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146)
* 4c5682b88e5c591d07223826f552502d85517001 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3644505763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c5682b88e5c591d07223826f552502d85517001 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147)
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617491018
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConverters.scala:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.common.schema.HoodieSchema
+import org.apache.spark.sql.types.DataType
+
+/**
+ * Allows conversion between HoodieSchema and Spark's Catalyst DataType.
+ *
+ * This trait provides version-agnostic interface for schema conversion,
+ * with version-specific implementations handling differences like TimestampNTZType.
+ */
+trait HoodieSchemaConverters {
Review Comment:
Will remove this as not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2620319288
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRecordContext.scala:
##########
@@ -53,15 +53,15 @@ trait SparkFileFormatInternalRecordContext extends BaseSparkInternalRecordContex
val schema = avroRecord.getSchema
val structType = HoodieInternalRowUtils.getCachedSchema(schema)
val deserializer = deserializerMap.getOrElseUpdate(schema, {
- sparkAdapter.createAvroDeserializer(schema, structType)
+ sparkAdapter.createAvroDeserializer(HoodieSchema.fromAvroSchema(schema), structType)
})
deserializer.deserialize(avroRecord).get.asInstanceOf[InternalRow]
}
override def convertToAvroRecord(record: InternalRow, schema: HoodieSchema): GenericRecord = {
val structType = HoodieInternalRowUtils.getCachedSchema(schema.toAvroSchema)
val serializer = serializerMap.getOrElseUpdate(schema.toAvroSchema, {
- sparkAdapter.createAvroSerializer(structType, schema.toAvroSchema, isNullable(schema.toAvroSchema))
+ sparkAdapter.createAvroSerializer(structType, schema, isNullable(schema.toAvroSchema))
Review Comment:
Can we simplify the `isNullable` to just use the `HoodieSchema`'s method directly?
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/TestHoodieMergeHandleWithSparkMerger.java:
##########
@@ -74,7 +74,7 @@
import static org.junit.jupiter.api.Assertions.assertTrue;
public class TestHoodieMergeHandleWithSparkMerger extends SparkClientFunctionalTestHarness {
- private static final Schema SCHEMA = getAvroSchema("AvroSchema", "AvroSchemaNS");
+ private static final HoodieSchema SCHEMA = getSchema("AvroSchema", "AvroSchemaNS");
Review Comment:
nitpick: remove `Avro` from the name and namespace
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieRelations.scala:
##########
@@ -40,13 +39,13 @@ class TestHoodieRelations {
"{\"name\":\"ts\",\"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"}],\"default\":null}," +
"{\"name\":\"partition\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
- val tableAvroSchema = new Schema.Parser().parse(avroSchemaString)
- val tableStructSchema = convertAvroSchemaToStructType(tableAvroSchema)
+ val tableAvroSchema = HoodieSchema.parse(avroSchemaString)
Review Comment:
Update variable names to avoid using `avro` to avoid confusion
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/TestHoodieMergeHandleWithSparkMerger.java:
##########
@@ -144,11 +144,11 @@ public static List<HoodieKey> getKeys(List<HoodieRecord> records) {
return records.stream().map(r -> r.getKey()).collect(Collectors.toList());
}
- private static Schema getAvroSchema(String schemaName, String schemaNameSpace) {
- return AvroConversionUtils.convertStructTypeToAvroSchema(SparkDatasetTestUtils.STRUCT_TYPE, schemaName, schemaNameSpace);
+ private static HoodieSchema getSchema(String schemaName, String schemaNameSpace) {
+ return HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(SparkDatasetTestUtils.STRUCT_TYPE, schemaName, schemaNameSpace);
}
- public HoodieWriteConfig getWriteConfig(Schema avroSchema, String recordMergerImplClass, String mergeStrategyId, RecordMergeMode recordMergeMode) {
+ public HoodieWriteConfig getWriteConfig(HoodieSchema hoodieSchema, String recordMergerImplClass, String mergeStrategyId, RecordMergeMode recordMergeMode) {
Review Comment:
For consistency, let's make the `hoodieSchema` simply `schema` when possible in all the classes.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -841,13 +840,13 @@ object HoodieBaseRelation extends SparkAdapterSupport {
val requiredRowSchema = requiredDataSchema.structTypeSchema
// NOTE: Schema has to be parsed at this point, since Avro's [[Schema]] aren't serializable
// to be passed from driver to executor
- val requiredAvroSchema = new Schema.Parser().parse(requiredDataSchema.avroSchemaStr)
- val avroToRowConverter = AvroConversionUtils.createAvroToInternalRowConverter(requiredAvroSchema, requiredRowSchema)
+ val requiredSchema = HoodieSchema.parse(requiredDataSchema.avroSchemaStr)
+ //TODO to fill impl for this
Review Comment:
Can you add some more details on the remaining work for this implementation? Should it be a separate ticket?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala:
##########
@@ -94,13 +95,13 @@ trait SparkAdapter extends Serializable {
* Creates instance of [[HoodieAvroSerializer]] providing for ability to serialize
* Spark's [[InternalRow]] into Avro payloads
*/
- def createAvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean): HoodieAvroSerializer
+ def createAvroSerializer(rootCatalystType: DataType, rootHoodieType: HoodieSchema, nullable: Boolean): HoodieAvroSerializer
Review Comment:
Instead of `rootHoodieType`, would it make more sense to have this as `rootTableType` or simply `rootType`?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
+ val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData.toAvroSchema()))
Review Comment:
Can we use `HoodieSchemaConversionUtils.convertHoodieSchemaToStructType` here?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestAvroSerDe.scala:
##########
@@ -51,8 +52,8 @@ class TestAvroSerDe extends SparkAdapterSupport {
val avroSchema = HoodieMetadataColumnStats.SCHEMA$
Review Comment:
Let's update this to be a `HoodieSchema` instead of converting it twice below
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlWithCustomKeyGenerator.scala:
##########
@@ -502,19 +502,19 @@ class TestSparkSqlWithCustomKeyGenerator extends HoodieSparkSqlTestBase {
testInserts(tableName, tsGenFunc, customPartitionFunc)
// Validate ts field is still of type int in the table
- validateTsFieldSchema(tablePath, "ts", Schema.Type.INT)
+ validateTsFieldSchema(tablePath, "ts", HoodieSchemaType.INT)
val metaClient = createMetaClient(spark, tablePath)
assertEquals(KeyGeneratorType.CUSTOM.getClassName, metaClient.getTableConfig.getKeyGeneratorClassName)
}
}
}
- private def validateTsFieldSchema(tablePath: String, fieldName: String, expectedType: Schema.Type): Unit = {
+ private def validateTsFieldSchema(tablePath: String, fieldName: String, expectedType: HoodieSchemaType): Unit = {
val metaClient = createMetaClient(spark, tablePath)
val schemaResolver = new TableSchemaResolver(metaClient)
- val nullableSchema = Schema.createUnion(Schema.create(Schema.Type.NULL), Schema.create(expectedType))
- assertEquals(nullableSchema, schemaResolver.getTableAvroSchema(true).getField(fieldName).schema())
+ val nullableSchema = HoodieSchema.createUnion(HoodieSchema.create(HoodieSchemaType.NULL), HoodieSchema.create(HoodieSchemaType.valueOf(expectedType.name())))
Review Comment:
there is a helper method for creating nullable schema that you can use here
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
Can we use the util instead?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -58,7 +56,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SparkSession, SQLContext}
import org.apache.spark.sql.HoodieCatalystExpressionUtils.{convertToCatalystExpression, generateUnsafeProjection}
-import org.apache.spark.sql.avro.HoodieSparkAvroSchemaConverters
+import org.apache.spark.sql.avro.{HoodieSparkAvroSchemaConverters, HoodieSparkSchemaConverters}
Review Comment:
Can the `HoodieSparkAvroSchemaConverters` be removed from this file as part of this PR?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.datasources.parquet
-import org.apache.hudi.{AvroConversionUtils, HoodieFileIndex, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSparkUtils, HoodieTableSchema, SparkAdapterSupport, SparkFileFormatInternalRowReaderContext}
+import org.apache.hudi.{AvroConversionUtils, HoodieFileIndex, HoodiePartitionCDCFileGroupMapping, HoodiePartitionFileSliceMapping, HoodieSchemaConversionUtils, HoodieSparkUtils, HoodieTableSchema, SparkAdapterSupport, SparkFileFormatInternalRowReaderContext}
Review Comment:
As part of this PR, lets track down the remaining usages of the `AvroConversionUtils` and make them use the `HoodieSchema` equivalent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3651620176
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a49e78d78fb9a210b6405236482916de1bc1a066 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218)
* e6a6deb8869158c0d0e18fc8e48d05009656c952 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3652257167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6a6deb8869158c0d0e18fc8e48d05009656c952 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226)
* 790babc0c83f242c3265bdbb5866e5b71230fb25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649707428
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188)
* e319753770fd6e626766024383138e3332729da9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3650134454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af3c3c76b7966506eec0e3eb22e565c13a34a770 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212)
* a49e78d78fb9a210b6405236482916de1bc1a066 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624828855
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -795,29 +793,30 @@ object HoodieBaseRelation extends SparkAdapterSupport {
* @param tableSchema schema to project (either of [[InternalSchema]] or Avro's [[Schema]])
* @param requiredColumns required top-level columns to be projected
*/
- def projectSchema(tableSchema: Either[Schema, InternalSchema], requiredColumns: Array[String]): (Schema, StructType, InternalSchema) = {
+ def projectSchema(tableSchema: Either[HoodieSchema, InternalSchema], requiredColumns: Array[String]): (HoodieSchema, StructType, InternalSchema) = {
tableSchema match {
case Right(internalSchema) =>
checkState(!internalSchema.isEmptySchema)
val prunedInternalSchema = InternalSchemaUtils.pruneInternalSchema(internalSchema, requiredColumns.toList.asJava)
- val requiredAvroSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema").toAvroSchema
- val requiredStructSchema = AvroConversionUtils.convertAvroSchemaToStructType(requiredAvroSchema)
+ val requiredSchema = InternalSchemaConverter.convert(prunedInternalSchema, "schema")
+ val requiredStructSchema = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(requiredSchema)
- (requiredAvroSchema, requiredStructSchema, prunedInternalSchema)
+ (requiredSchema, requiredStructSchema, prunedInternalSchema)
- case Left(avroSchema) =>
- val fieldMap = avroSchema.getFields.asScala.map(f => f.name() -> f).toMap
+ case Left(hoodieSchema) =>
+ val fieldMap = hoodieSchema.getFields.asScala.map(f => f.name() -> f).toMap
val requiredFields = requiredColumns.map { col =>
- val f = fieldMap(col)
- // We have to create a new [[Schema.Field]] since Avro schemas can't share field
- // instances (and will throw "org.apache.avro.AvroRuntimeException: Field already used")
- createNewSchemaField(f.name(), f.schema(), f.doc(), f.defaultVal(), f.order())
Review Comment:
have now used `HoodieSchemaUtils.createNewSchemaField`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632586381
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroDeserializer.scala:
##########
@@ -17,15 +17,16 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
+class HoodieSpark3_3AvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType)
Review Comment:
@yihua mentioned to keep these as avro schemas and only the adapter needs to be changed for each spark version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2623543599
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala:
##########
@@ -563,15 +563,15 @@ object ExpressionPayload {
}
}
- private def mergeSchema(a: Schema, b: Schema): Schema = {
+ private def mergeSchema(a: HoodieSchema, b: HoodieSchema): HoodieSchema = {
val mergedFields =
a.getFields.asScala.map(field =>
- createNewSchemaField("source_" + field.name,
- field.schema, field.doc, field.defaultVal, field.order)) ++
+ HoodieSchemaField.of("source_" + field.name(),
Review Comment:
Let's use the util here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631609669
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionHelper.scala:
##########
@@ -41,11 +42,11 @@ class TestAvroConversionHelper extends FunSuite with Matchers {
val dateInputData = Seq(7, 365, 0)
test("Logical type: date") {
- val schema = new Schema.Parser().parse(dateSchema)
- val convertor = AvroConversionUtils.createConverterToRow(schema, AvroConversionUtils.convertAvroSchemaToStructType(schema))
+ val schema = HoodieSchema.parse(dateSchema)
+ val convertor = AvroConversionUtils.createConverterToRow(schema.toAvroSchema, HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(schema))
Review Comment:
I can add a unit test for `createGenericRecordToInternalRowConverter` similar to the function its based on `createAvroToInternalRowConverter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672221064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18706cf60f015018b214ea78997b17894198021a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662444388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
* c7dcdab5a769970399cdd4c29ea76f75335c50d4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296)
* b181d4bf2a4da4c5b54a5a5f192555af659c4f7c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3671911789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
* 18706cf60f015018b214ea78997b17894198021a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672766244
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e66b1f84650709425024b101fe319b5c547d3746 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628769708
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -369,15 +367,15 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
val fileSplits = collectFileSplits(partitionFilters, dataFilters)
- val tableAvroSchemaStr = tableAvroSchema.toString
+ val tableSchemaStr = tableSchema.toString
- val tableSchema = HoodieTableSchema(tableStructSchema, tableAvroSchemaStr, internalSchemaOpt)
+ val schema = HoodieTableSchema(tableStructSchema, tableSchemaStr, internalSchemaOpt)
Review Comment:
Could we directly store `HoodieSchema` in `HoodieTableSchema`?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -369,15 +367,15 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
val fileSplits = collectFileSplits(partitionFilters, dataFilters)
- val tableAvroSchemaStr = tableAvroSchema.toString
+ val tableSchemaStr = tableSchema.toString
- val tableSchema = HoodieTableSchema(tableStructSchema, tableAvroSchemaStr, internalSchemaOpt)
+ val schema = HoodieTableSchema(tableStructSchema, tableSchemaStr, internalSchemaOpt)
Review Comment:
Change `avroSchemaStr` to `tableSchemaStr` in `HoodieTableSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3673411821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e66b1f84650709425024b101fe319b5c547d3746 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359)
* c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3675129516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367)
* 452d8869cb0deeef79738582648888e57da74584 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2635514355
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,9 +210,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val exclusionFields = new java.util.HashSet[String]()
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
- val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
Review Comment:
My bad, let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672494747
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18706cf60f015018b214ea78997b17894198021a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354)
* df79b71c60f836a1069424edcc0f31ac282c1f2a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356)
* e66b1f84650709425024b101fe319b5c547d3746 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3672499263
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df79b71c60f836a1069424edcc0f31ac282c1f2a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356)
* e66b1f84650709425024b101fe319b5c547d3746 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2633385786
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCDataFrameSuite.scala:
##########
@@ -21,6 +21,7 @@ package org.apache.hudi.functional.cdc
import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.DataSourceWriteOptions.{MOR_TABLE_TYPE_OPT_VAL, PARTITIONPATH_FIELD_OPT_KEY, PRECOMBINE_FIELD_OPT_KEY, RECORDKEY_FIELD_OPT_KEY}
import org.apache.hudi.QuickstartUtils.getQuickstartWriteConfigs
+import org.apache.hudi.common.schema.HoodieSchema
Review Comment:
nit: This import is redundant
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroSerializer.scala:
##########
@@ -17,13 +17,14 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroSerializer(rootCatalystType: DataType, rootAvroType: Schema, nullable: Boolean)
+class HoodieSpark3_3AvroSerializer(rootCatalystType: DataType, rootType: HoodieSchema, nullable: Boolean)
Review Comment:
Same here
##########
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_5AvroSerializer.scala:
##########
@@ -17,6 +17,8 @@
package org.apache.spark.sql.avro
+import org.apache.hudi.common.schema.HoodieSchema
Review Comment:
nit: import is unnecessary
##########
hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_3AvroDeserializer.scala:
##########
@@ -17,15 +17,16 @@
package org.apache.spark.sql.avro
-import org.apache.avro.Schema
+import org.apache.hudi.common.schema.HoodieSchema
+
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
import org.apache.spark.sql.types.DataType
-class HoodieSpark3_3AvroDeserializer(rootAvroType: Schema, rootCatalystType: DataType)
+class HoodieSpark3_3AvroDeserializer(rootType: HoodieSchema, rootCatalystType: DataType)
Review Comment:
Look like the logic in Spark 3.3 is not fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2635099057
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -212,8 +211,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
- val requestedAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(requestedSchema, sanitizedTableName), exclusionFields)
- val dataAvroSchema = AvroSchemaUtils.pruneDataSchema(avroTableSchema, AvroConversionUtils.convertStructTypeToAvroSchema(dataSchema, sanitizedTableName), exclusionFields)
+ val requestedHoodieSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(requestedSchema, sanitizedTableName), exclusionFields)
+ val dataHoodieSchema = HoodieSchemaUtils.pruneDataSchema(schema, HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(dataSchema, sanitizedTableName), exclusionFields)
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDDV2.scala:
##########
@@ -193,8 +192,8 @@ class HoodieMergeOnReadRDDV2(@transient sc: SparkContext,
.withBaseFileOption(baseFileOption)
.withPartitionPath(partitionPath)
.withProps(properties)
- .withDataSchema(HoodieSchema.parse(tableSchema.avroSchemaStr))
- .withRequestedSchema(HoodieSchema.parse(requiredSchema.avroSchemaStr))
+ .withDataSchema(tableSchema.tableSchema)
+ .withRequestedSchema(requiredSchema.tableSchema)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3676158265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
}, {
"hash" : "18706cf60f015018b214ea78997b17894198021a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10354",
"triggerID" : "18706cf60f015018b214ea78997b17894198021a",
"triggerType" : "PUSH"
}, {
"hash" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10356",
"triggerID" : "df79b71c60f836a1069424edcc0f31ac282c1f2a",
"triggerType" : "PUSH"
}, {
"hash" : "e66b1f84650709425024b101fe319b5c547d3746",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10359",
"triggerID" : "e66b1f84650709425024b101fe319b5c547d3746",
"triggerType" : "PUSH"
}, {
"hash" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10367",
"triggerID" : "c10a3dfabb5337ceaa56b26e8d93e30aadeb99aa",
"triggerType" : "PUSH"
}, {
"hash" : "452d8869cb0deeef79738582648888e57da74584",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10372",
"triggerID" : "452d8869cb0deeef79738582648888e57da74584",
"triggerType" : "PUSH"
}, {
"hash" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10373",
"triggerID" : "b9f300a7981e88da2c623aa08fd47e190a6f3374",
"triggerType" : "PUSH"
}, {
"hash" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374",
"triggerID" : "a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4c7d06e7f3b6f97c1df83b20b2c2d690720e3ae Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10374)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17573:
URL: https://github.com/apache/hudi/pull/17573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2635402136
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -211,9 +210,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath: String,
val exclusionFields = new java.util.HashSet[String]()
exclusionFields.add("op")
partitionSchema.fields.foreach(f => exclusionFields.add(f.name))
- val requestedSchema = StructType(requiredSchema.fields ++ partitionSchema.fields.filter(f => mandatoryFields.contains(f.name)))
Review Comment:
This struct type schema needs to be kept and renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3661060711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e82c5dd3f3374e1806f3d358379f9f4e85f174a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283)
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3661052356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e82c5dd3f3374e1806f3d358379f9f4e85f174a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283)
* 185d596831f4eefc36a9236eaae9d8466cf0bb2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3656798498
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3db175289bb9a54c723b7ea5972c46c81300c0fe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3647956016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
* b766aed2cb16e04408b1fba1a83f8673177836be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184)
* 2ed5babfa21228fbd4b56d2706ef5ddfdaac980f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3647758357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149)
* b766aed2cb16e04408b1fba1a83f8673177836be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3650133269
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af3c3c76b7966506eec0e3eb22e565c13a34a770 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212)
* a49e78d78fb9a210b6405236482916de1bc1a066 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649726948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e319753770fd6e626766024383138e3332729da9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208)
* c9377b9fadb42c5ee11099fb01c5cd53f13f8120 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2617423850
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableChangeColumnCommand.scala:
##########
@@ -86,10 +85,10 @@ case class AlterHoodieTableChangeColumnCommand(
Seq.empty[Row]
}
- private def validateSchema(newSchema: Schema, metaClient: HoodieTableMetaClient): Unit = {
+ private def validateSchema(newSchema: HoodieSchema, metaClient: HoodieTableMetaClient): Unit = {
val schemaUtil = new TableSchemaResolver(metaClient)
- val tableSchema = HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchema(false))
- if (!AvroSchemaUtils.isSchemaCompatible(tableSchema, newSchema)) {
+ val tableSchema = HoodieSchemaUtils.createHoodieWriteSchema(schemaUtil.getTableSchema(false).toString, false)
Review Comment:
Currently the util only takes in a string
<img width="1302" height="484" alt="Screenshot 2025-12-14 at 11 29 15 AM" src="https://github.com/user-attachments/assets/775a2a13-b66f-45da-afba-8bbfe2c9cae6" />
Is the ask then to just make a util that directly operates on the HoodieSchema object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3649797648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9377b9fadb42c5ee11099fb01c5cd53f13f8120 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211)
* af3c3c76b7966506eec0e3eb22e565c13a34a770 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628651565
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -595,7 +674,29 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
public static HoodieSchema projectSchema(HoodieSchema fileSchema, List<String> fields) {
return HoodieSchema.fromAvroSchema(HoodieAvroUtils.projectSchema(fileSchema.toAvroSchema(), fields));
}
+
+ /**
+ * Gets the fully-qualified Avro record name for a Hudi table.
+ * This is equivalent to {@link AvroSchemaUtils#getAvroRecordQualifiedName(String)}
+ * but provides a HoodieSchema-context API.
+ *
+ * <p>The qualified name follows the pattern: hoodie.{tableName}.{tableName}_record
+ * where tableName is sanitized for Avro compatibility.</p>
+ *
+ * @param tableName the Hudi table name
+ * @return the fully-qualified Avro record name (e.g., "hoodie.my_table.my_table_record")
+ * @throws IllegalArgumentException if tableName is null or empty
+ * @since 1.2.0
+ */
+ public static String getRecordQualifiedName(String tableName) {
+ ValidationUtils.checkArgument(tableName != null && !tableName.trim().isEmpty(),
+ "Table name cannot be null or empty");
+
+ // Delegate to AvroSchemaUtils
+ return AvroSchemaUtils.getAvroRecordQualifiedName(tableName);
Review Comment:
Should we move the actual logic here as the logic is not really Avro specific?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSchemaConversionUtils.scala:
##########
@@ -18,10 +18,15 @@
package org.apache.hudi
Review Comment:
This class should be under `org.apache.hudi.spark.schema` ?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
Review Comment:
Should we git rid of this method which does not take the `defaultVal` argument?
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -307,6 +325,31 @@ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema s
return HoodieSchemaField.fromAvroField(avroField);
}
+ /**
+ * Creates a new schema field with the specified properties, including field order.
+ * This is equivalent to HoodieAvroUtils.createNewSchemaField() but returns HoodieSchemaField.
+ *
+ * @param name field name
+ * @param schema field schema
+ * @param doc field documentation (can be null)
+ * @param defaultValue default value (can be null)
+ * @param order field order for sorting
+ * @return new HoodieSchemaField instance
+ * @throws IllegalArgumentException if name, schema, or order is null/empty
+ * @since 1.2.0
+ */
+ public static HoodieSchemaField createNewSchemaField(String name, HoodieSchema schema,
+ String doc, Object defaultValue, HoodieFieldOrder order) {
+ ValidationUtils.checkArgument(name != null && !name.isEmpty(), "Field name cannot be null or empty");
+ ValidationUtils.checkArgument(schema != null, "Field schema cannot be null");
+ ValidationUtils.checkArgument(order != null, "Field order cannot be null");
+
+ // Delegate to HoodieAvroUtils
+ Schema.Field avroField = HoodieAvroUtils.createNewSchemaField(
+ name, schema.toAvroSchema(), doc, defaultValue, order.toAvroOrder());
+ return HoodieSchemaField.fromAvroField(avroField);
+ }
Review Comment:
Can `createNewSchemaField(String name, HoodieSchema schema, String doc, Object defaultValue)` reuse the logic here?
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -79,6 +79,24 @@ public static HoodieSchema createHoodieWriteSchema(String schema, boolean withOp
return HoodieSchema.fromAvroSchema(avroSchema);
}
+ /**
+ * Creates a write schema for Hudi operations, adding necessary metadata fields.
+ * This is equivalent to HoodieAvroUtils.createHoodieWriteSchema() but operates on HoodieSchema.
+ *
+ * @param schema the base HoodieSchema
+ * @param withOperationField whether to include operation metadata field
+ * @return HoodieSchema configured for write operations with metadata fields added
+ * @throws IllegalArgumentException if schema is null
+ * @since 1.2.0
+ */
+ public static HoodieSchema createHoodieWriteSchema(HoodieSchema schema, boolean withOperationField) {
+ ValidationUtils.checkArgument(schema != null, "Schema cannot be null");
+
+ // Convert to Avro, delegate to existing utility, convert back
+ Schema avroSchema = HoodieAvroUtils.addMetadataFields(schema.toAvroSchema(), withOperationField);
Review Comment:
Should we have a util to add meta fields to `HoodieSchema`? Looks like `HoodieSchemaUtils#addMetadataFields` or `HoodieSchema#addMetadataFields` already does the same (can we also remove the duplicate method)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628659457
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestBucketIndexSupport.scala:
##########
@@ -66,7 +66,7 @@ class TestBucketIndexSupport extends HoodieSparkClientTestBase with PredicateHel
)
)
- val avroSchema = new Schema.Parser().parse(avroSchemaStr)
+ val hoodieSchema = HoodieSchema.parse(avroSchemaStr)
Review Comment:
nit: should we name this as `schema` for brevity?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSourceDefaults.scala:
##########
@@ -625,14 +624,14 @@ class TestDataSourceDefaults extends ScalaAssertionSupport {
val props = new TypedProperties()
OrderingFieldsTestUtils.setOrderingFieldsConfig(props, key, "favoriteIntNumber")
val baseOrderingVal: Object = baseRecord.get("favoriteIntNumber")
- val fieldSchema: Schema = baseRecord.getSchema().getField("favoriteIntNumber").schema()
+ val fieldSchema: HoodieSchema = HoodieSchema.fromAvroSchema(baseRecord.getSchema().getField("favoriteIntNumber").schema())
- val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
+ val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
val laterRecord = SchemaTestUtil
.generateAvroRecordFromJson(schema, 2, "001", "f1")
val laterOrderingVal: Object = laterRecord.get("favoriteIntNumber")
- val newerPayload = new OverwriteWithLatestAvroPayload(laterRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, laterOrderingVal, false).asInstanceOf[Comparable[_]])
+ val newerPayload = new OverwriteWithLatestAvroPayload(laterRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, laterOrderingVal, false).asInstanceOf[Comparable[_]])
Review Comment:
Same here and other places
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionBucketIndexSupport.scala:
##########
@@ -67,19 +67,19 @@ class TestPartitionBucketIndexSupport extends TestBucketIndexSupport {
// init
val testKeyGenerator = new NonpartitionedKeyGenerator(configProperties)
- var record = new GenericData.Record(avroSchema)
+ var record = new GenericData.Record(hoodieSchema.toAvroSchema)
Review Comment:
nit: same on `schema` instead of `hoodieSchema`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestColumnStatsIndex.scala:
##########
@@ -1101,8 +1100,8 @@ class TestColumnStatsIndex extends ColumnStatIndexTestBase {
def testDeserialize(description: String, expected: JBigDecimal, wrapperCreator: WrapperCreator): Unit = {
val dt = DecimalType(10, 2)
// Get the schema from the DecimalWrapper's Avro definition.
- val schema: Schema = DecimalWrapper.SCHEMA$.getField("value").schema()
- val wrapper = wrapperCreator.create(expected, schema)
+ val schema: HoodieSchema = HoodieSchema.fromAvroSchema(DecimalWrapper.SCHEMA$.getField("value").schema())
+ val wrapper = wrapperCreator.create(expected, schema.toAvroSchema)
Review Comment:
nit: these changes seem unnecessary as they convert the schema back and forth
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSourceDefaults.scala:
##########
@@ -625,14 +624,14 @@ class TestDataSourceDefaults extends ScalaAssertionSupport {
val props = new TypedProperties()
OrderingFieldsTestUtils.setOrderingFieldsConfig(props, key, "favoriteIntNumber")
val baseOrderingVal: Object = baseRecord.get("favoriteIntNumber")
- val fieldSchema: Schema = baseRecord.getSchema().getField("favoriteIntNumber").schema()
+ val fieldSchema: HoodieSchema = HoodieSchema.fromAvroSchema(baseRecord.getSchema().getField("favoriteIntNumber").schema())
- val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
+ val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
Review Comment:
nit: no need of `org.apache.hudi.common.schema.` prefix
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionHelper.scala:
##########
@@ -41,11 +42,11 @@ class TestAvroConversionHelper extends FunSuite with Matchers {
val dateInputData = Seq(7, 365, 0)
test("Logical type: date") {
- val schema = new Schema.Parser().parse(dateSchema)
- val convertor = AvroConversionUtils.createConverterToRow(schema, AvroConversionUtils.convertAvroSchemaToStructType(schema))
+ val schema = HoodieSchema.parse(dateSchema)
+ val convertor = AvroConversionUtils.createConverterToRow(schema.toAvroSchema, HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(schema))
Review Comment:
Is there any test on `createGenericRecordToInternalRowConverter`? If not, should we add add another test to use `createGenericRecordToInternalRowConverter` for testing the conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2628693995
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriterWithTestFormat.scala:
##########
@@ -268,8 +268,8 @@ class TestHoodieSparkSqlWriterWithTestFormat extends HoodieSparkWriterTestBase {
.updated(HoodieTableConfig.TABLE_FORMAT.key, "test-format")
// generate the inserts
- val schema = DataSourceTestUtils.getStructTypeExampleSchema
- val structType = AvroConversionUtils.convertAvroSchemaToStructType(schema)
+ val schema = DataSourceTestUtils.getStructTypeExampleHoodieSchema
Review Comment:
Could we change `DataSourceTestUtils#getStructTypeExampleSchema` directly instead of adding a new one `getStructTypeExampleHoodieSchema`?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
Review Comment:
Should `TableSchemaResolver` provide a new API `getTableSchemaFromDataFile` that returns `HoodieSchema`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestTableSchemaResolverWithSparkSQL.scala:
##########
@@ -122,10 +122,11 @@ class TestTableSchemaResolverWithSparkSQL extends HoodieSparkWriterTestBase {
metaClient.reloadActiveTimeline()
var tableSchemaResolverParsingException: Exception = null
try {
- val schemaFromData = new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile
- val structFromData = AvroConversionUtils.convertAvroSchemaToStructType(HoodieAvroUtils.removeMetadataFields(schemaFromData))
- val schemeDesign = new Schema.Parser().parse(schemaString)
- val structDesign = AvroConversionUtils.convertAvroSchemaToStructType(schemeDesign)
+ val schemaFromData = HoodieSchema.fromAvroSchema(
+ new TableSchemaResolver(metaClient).getTableAvroSchemaFromDataFile)
+ val structFromData = HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(org.apache.hudi.common.schema.HoodieSchemaUtils.removeMetadataFields(schemaFromData))
Review Comment:
nit: import `org.apache.hudi.common.schema.HoodieSchemaUtils`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2624411003
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala:
##########
@@ -483,15 +483,15 @@ object ExpressionPayload {
avroDeserializerCache.get()
.get(schema, new Function[Schema, HoodieAvroDeserializer] {
override def apply(t: Schema): HoodieAvroDeserializer =
- sparkAdapter.createAvroDeserializer(schema, convertAvroSchemaToStructType(schema))
+ sparkAdapter.createAvroDeserializer(HoodieSchema.fromAvroSchema(schema), convertHoodieSchemaToDataType(HoodieSchema.fromAvroSchema(schema)))
})
}
private def getAvroSerializerFor(schema: Schema) = {
Review Comment:
Will try to make this change for this class only to have caches take in HoodieSchema. However i think my concern is this might cascade and touch other classes potentially(will give first attempt).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3662948855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28ba7fd754f34deb55540d521795e15f48fc5ae2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3663008078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24dda257806ad8dae89de72faf95f04e02242399 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631550917
##########
hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java:
##########
@@ -595,7 +674,29 @@ public static HoodieSchema getRecordKeySchema() {
* @param fields list of fields to project
* @return projected schema containing only specified fields
*/
+
public static HoodieSchema projectSchema(HoodieSchema fileSchema, List<String> fields) {
return HoodieSchema.fromAvroSchema(HoodieAvroUtils.projectSchema(fileSchema.toAvroSchema(), fields));
}
+
+ /**
+ * Gets the fully-qualified Avro record name for a Hudi table.
+ * This is equivalent to {@link AvroSchemaUtils#getAvroRecordQualifiedName(String)}
+ * but provides a HoodieSchema-context API.
+ *
+ * <p>The qualified name follows the pattern: hoodie.{tableName}.{tableName}_record
+ * where tableName is sanitized for Avro compatibility.</p>
+ *
+ * @param tableName the Hudi table name
+ * @return the fully-qualified Avro record name (e.g., "hoodie.my_table.my_table_record")
+ * @throws IllegalArgumentException if tableName is null or empty
+ * @since 1.2.0
+ */
+ public static String getRecordQualifiedName(String tableName) {
+ ValidationUtils.checkArgument(tableName != null && !tableName.trim().isEmpty(),
+ "Table name cannot be null or empty");
+
+ // Delegate to AvroSchemaUtils
+ return AvroSchemaUtils.getAvroRecordQualifiedName(tableName);
Review Comment:
I am not quite sure on what you mean by this comment? I think we can just keep this to be 1-1 with avro util for now unless I am missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631960238
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkSortAndSizeClustering.java:
##########
@@ -23,12 +23,12 @@
import org.apache.hudi.avro.model.HoodieClusteringPlan;
import org.apache.hudi.client.WriteStatus;
import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.schema.HoodieSchema;
Review Comment:
will fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17573:
URL: https://github.com/apache/hudi/pull/17573#issuecomment-3671601137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10143",
"triggerID" : "204f6219585adb778e51680eaf3bcedeb8e8d178",
"triggerType" : "PUSH"
}, {
"hash" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10146",
"triggerID" : "8b3fff68c0346d4464e75e29b62578dca730043a",
"triggerType" : "PUSH"
}, {
"hash" : "4c5682b88e5c591d07223826f552502d85517001",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10147",
"triggerID" : "4c5682b88e5c591d07223826f552502d85517001",
"triggerType" : "PUSH"
}, {
"hash" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10149",
"triggerID" : "e8c3120ab1a5b469b0b6497a1d18fedac1ff9d3f",
"triggerType" : "PUSH"
}, {
"hash" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10184",
"triggerID" : "b766aed2cb16e04408b1fba1a83f8673177836be",
"triggerType" : "PUSH"
}, {
"hash" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10188",
"triggerID" : "2ed5babfa21228fbd4b56d2706ef5ddfdaac980f",
"triggerType" : "PUSH"
}, {
"hash" : "e319753770fd6e626766024383138e3332729da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10208",
"triggerID" : "e319753770fd6e626766024383138e3332729da9",
"triggerType" : "PUSH"
}, {
"hash" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10211",
"triggerID" : "c9377b9fadb42c5ee11099fb01c5cd53f13f8120",
"triggerType" : "PUSH"
}, {
"hash" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10212",
"triggerID" : "af3c3c76b7966506eec0e3eb22e565c13a34a770",
"triggerType" : "PUSH"
}, {
"hash" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10218",
"triggerID" : "a49e78d78fb9a210b6405236482916de1bc1a066",
"triggerType" : "PUSH"
}, {
"hash" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10226",
"triggerID" : "e6a6deb8869158c0d0e18fc8e48d05009656c952",
"triggerType" : "PUSH"
}, {
"hash" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10229",
"triggerID" : "790babc0c83f242c3265bdbb5866e5b71230fb25",
"triggerType" : "PUSH"
}, {
"hash" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10243",
"triggerID" : "3db175289bb9a54c723b7ea5972c46c81300c0fe",
"triggerType" : "PUSH"
}, {
"hash" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10257",
"triggerID" : "40f3c37722c93a9a37136f128ab5fc135feb34b9",
"triggerType" : "PUSH"
}, {
"hash" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10260",
"triggerID" : "a5500fd7a2d0676063d950f5e51cee10ebbe1e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10283",
"triggerID" : "8e82c5dd3f3374e1806f3d358379f9f4e85f174a",
"triggerType" : "PUSH"
}, {
"hash" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10289",
"triggerID" : "185d596831f4eefc36a9236eaae9d8466cf0bb2d",
"triggerType" : "PUSH"
}, {
"hash" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10296",
"triggerID" : "c7dcdab5a769970399cdd4c29ea76f75335c50d4",
"triggerType" : "PUSH"
}, {
"hash" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10302",
"triggerID" : "b181d4bf2a4da4c5b54a5a5f192555af659c4f7c",
"triggerType" : "PUSH"
}, {
"hash" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10303",
"triggerID" : "28ba7fd754f34deb55540d521795e15f48fc5ae2",
"triggerType" : "PUSH"
}, {
"hash" : "24dda257806ad8dae89de72faf95f04e02242399",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10304",
"triggerID" : "24dda257806ad8dae89de72faf95f04e02242399",
"triggerType" : "PUSH"
}, {
"hash" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306",
"triggerID" : "ed494441741a5aee888a3a384acbee9d033a19ac",
"triggerType" : "PUSH"
}, {
"hash" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352",
"triggerID" : "e7b1643815be3c3dac004de710bb9343658f9d66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed494441741a5aee888a3a384acbee9d033a19ac Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10306)
* e7b1643815be3c3dac004de710bb9343658f9d66 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2632579394
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSourceDefaults.scala:
##########
@@ -625,14 +624,14 @@ class TestDataSourceDefaults extends ScalaAssertionSupport {
val props = new TypedProperties()
OrderingFieldsTestUtils.setOrderingFieldsConfig(props, key, "favoriteIntNumber")
val baseOrderingVal: Object = baseRecord.get("favoriteIntNumber")
- val fieldSchema: Schema = baseRecord.getSchema().getField("favoriteIntNumber").schema()
+ val fieldSchema: HoodieSchema = HoodieSchema.fromAvroSchema(baseRecord.getSchema().getField("favoriteIntNumber").schema())
- val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, HoodieAvroUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
+ val basePayload = new OverwriteWithLatestAvroPayload(baseRecord, org.apache.hudi.common.schema.HoodieSchemaUtils.convertValueForSpecificDataTypes(fieldSchema, baseOrderingVal, false).asInstanceOf[Comparable[_]])
Review Comment:
@yihua recommend to use scala import to give a different to avoid using a package prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(schema): Migrate spark reader side related classes to use HoodieSchema directly [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #17573:
URL: https://github.com/apache/hudi/pull/17573#discussion_r2631339320
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
Review Comment:
<img width="763" height="188" alt="Screenshot 2025-12-18 at 6 20 30 AM" src="https://github.com/user-attachments/assets/decefdb9-a69d-4807-a01f-06c396739f66" />
There are no production usages of it, so it can be removed. I think when looking at HoodieAvroUtils we also did not have any three arg constructor like this so I am thinking it might be less confusing and have the caller be explicit on what they want default val as.
Followed Ethan rec here
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -102,7 +102,14 @@ object HoodieSparkSchemaConverters {
val fields = st.map { f =>
val fieldSchema = toHoodieType(f.dataType, f.nullable, f.name, childNameSpace)
val doc = f.getComment.orNull
- HoodieSchemaField.of(f.name, fieldSchema, doc)
Review Comment:
<img width="763" height="188" alt="Screenshot 2025-12-18 at 6 20 30 AM" src="https://github.com/user-attachments/assets/decefdb9-a69d-4807-a01f-06c396739f66" />
There are no production usages of it, so it can be removed. I think when looking at HoodieAvroUtils we also did not have any three arg constructor like this so I am thinking it might be less confusing and have the caller be explicit on what they want default val as.
Update: have made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org