You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/12/08 18:20:11 UTC
[PR] [HUDI-7201] use target schema if source is empty [hudi]
jonvex opened a new pull request, #10288:
URL: https://github.com/apache/hudi/pull/10288
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #10288:
URL: https://github.com/apache/hudi/pull/10288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1847914479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"triggerType" : "PUSH"
}, {
"hash" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21412",
"triggerID" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 661f97f45b7705bb2e8b24e11e3c6a510b3b05ec UNKNOWN
* 0dad9618fd72f87aeac04ea0104e4d8127799ce5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1849037426
<img width="1266" alt="image" src="https://github.com/apache/hudi/assets/513218/d570388c-afe2-48b0-b464-8aae82c2fc47">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #10288:
URL: https://github.com/apache/hudi/pull/10288#discussion_r1421675446
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaEvolutionUtils.java:
##########
@@ -140,10 +140,14 @@ public static Schema reconcileSchema(Schema incomingSchema, Schema oldTableSchem
* @return schema (based off {@code source} one) that has nullability constraints and datatypes reconciled
*/
public static Schema reconcileSchemaRequirements(Schema sourceSchema, Schema targetSchema, Map<String, String> opts) {
- if (sourceSchema.getType() == Schema.Type.NULL || sourceSchema.getFields().isEmpty() || targetSchema.getFields().isEmpty()) {
+ if (targetSchema.getType() == Schema.Type.NULL || targetSchema.getFields().isEmpty()) {
Review Comment:
if its empty df i.e. 0 records, its ok to go ahead and make an empty commit w/ schema set to target schema. So, I don't see much necessity. Let me know wdyt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #10288:
URL: https://github.com/apache/hudi/pull/10288#discussion_r1421097099
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaEvolutionUtils.java:
##########
@@ -140,10 +140,14 @@ public static Schema reconcileSchema(Schema incomingSchema, Schema oldTableSchem
* @return schema (based off {@code source} one) that has nullability constraints and datatypes reconciled
*/
public static Schema reconcileSchemaRequirements(Schema sourceSchema, Schema targetSchema, Map<String, String> opts) {
- if (sourceSchema.getType() == Schema.Type.NULL || sourceSchema.getFields().isEmpty() || targetSchema.getFields().isEmpty()) {
+ if (targetSchema.getType() == Schema.Type.NULL || targetSchema.getFields().isEmpty()) {
Review Comment:
This will also be problematic when transformation schema is not fully compatible with source schema. Can we take the approach if the input df is empty, do not set schema ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1847697530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"triggerType" : "PUSH"
}, {
"hash" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21412",
"triggerID" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 661f97f45b7705bb2e8b24e11e3c6a510b3b05ec UNKNOWN
* 0dad9618fd72f87aeac04ea0104e4d8127799ce5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1847654228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"triggerType" : "PUSH"
}, {
"hash" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0dad9618fd72f87aeac04ea0104e4d8127799ce5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 661f97f45b7705bb2e8b24e11e3c6a510b3b05ec UNKNOWN
* 0dad9618fd72f87aeac04ea0104e4d8127799ce5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "zyclove (via GitHub)" <gi...@apache.org>.
zyclove commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1849234079
@nsivabalan @danny0405
Seeing that there are still 34 issues to be fixed in release 0.14.1, we are very looking forward to it.
We hope it can be speeded up. We also hope to switch the prod business to version 0.14.1 as soon as possible.
It is best to be done before the year end. Version 0.14.1 should be the best and most complete version ever.
I am also looking forward to the official version 1.0. thanks everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10288:
URL: https://github.com/apache/hudi/pull/10288#issuecomment-1847645692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "661f97f45b7705bb2e8b24e11e3c6a510b3b05ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 661f97f45b7705bb2e8b24e11e3c6a510b3b05ec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7201] Schema Evolution: use target schema if source is empty [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #10288:
URL: https://github.com/apache/hudi/pull/10288#discussion_r1421674974
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaEvolutionUtils.java:
##########
@@ -140,10 +140,14 @@ public static Schema reconcileSchema(Schema incomingSchema, Schema oldTableSchem
* @return schema (based off {@code source} one) that has nullability constraints and datatypes reconciled
*/
public static Schema reconcileSchemaRequirements(Schema sourceSchema, Schema targetSchema, Map<String, String> opts) {
- if (sourceSchema.getType() == Schema.Type.NULL || sourceSchema.getFields().isEmpty() || targetSchema.getFields().isEmpty()) {
+ if (targetSchema.getType() == Schema.Type.NULL || targetSchema.getFields().isEmpty()) {
Review Comment:
Balaji: I guess up until 0.14.0, we had isempty check explicitly for df that we consume from source before ingesting to hudi. but due to perf reasons, we removed that check. Hence we are running into this issue post 0.14.0
So, would prefer to avoid checking df.isEmpty in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org