You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2024/01/23 04:53:50 UTC
[PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
the-other-tim-brown opened a new pull request, #10549:
URL: https://github.com/apache/hudi/pull/10549
### Change Logs
Updates the `Transformer` classes to take in a `Supplier<Option<Schema>>` instead of `Option<Schema>` so that the latest value can be used when the `transformedSchema` method is called.
### Impact
- Provides more accurate schema
- Lazy evaluation of the schema also avoids an extra call to schema registries and potential exceptions that come with that
### Risk level (write none, low medium or high below)
None, this code is not actively used
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1908945912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
}, {
"hash" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104)
* ca627db36503a81c4223edde799bd344b9cf2b05 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1908955641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
}, {
"hash" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22148",
"triggerID" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104)
* ca627db36503a81c4223edde799bd344b9cf2b05 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22148)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #10549:
URL: https://github.com/apache/hudi/pull/10549#discussion_r1465518543
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -120,6 +121,7 @@ private void validateIdentifier(String id, Set<String> identifiers, String confi
private StructType getExpectedTransformedSchema(TransformerInfo transformerInfo, JavaSparkContext jsc, SparkSession sparkSession,
Option<StructType> incomingStructOpt, Option<Dataset<Row>> rowDatasetOpt, TypedProperties properties) {
+ Option<Schema> sourceSchemaOpt = sourceSchemaSupplier.get();
Review Comment:
added a test to validate that the supplier is called per invocation of the method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1905315665
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10549:
URL: https://github.com/apache/hudi/pull/10549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10549:
URL: https://github.com/apache/hudi/pull/10549#discussion_r1465442627
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -120,6 +121,7 @@ private void validateIdentifier(String id, Set<String> identifiers, String confi
private StructType getExpectedTransformedSchema(TransformerInfo transformerInfo, JavaSparkContext jsc, SparkSession sparkSession,
Option<StructType> incomingStructOpt, Option<Dataset<Row>> rowDatasetOpt, TypedProperties properties) {
+ Option<Schema> sourceSchemaOpt = sourceSchemaSupplier.get();
Review Comment:
Could you add a test of a scenario where the schema is evolved from the schema provider, and the `getExpectedTransformedSchema` returns the updated `StructType` instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1905322353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1905529795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7323] Use a schema supplier instead of a static value [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10549:
URL: https://github.com/apache/hudi/pull/10549#issuecomment-1909151475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22104",
"triggerID" : "ee8ed782107e9ef4aa7ebe50fa22fc68c6c14602",
"triggerType" : "PUSH"
}, {
"hash" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22148",
"triggerID" : "ca627db36503a81c4223edde799bd344b9cf2b05",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca627db36503a81c4223edde799bd344b9cf2b05 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22148)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org