You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2024/01/23 19:23:11 UTC
[PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
jonvex opened a new pull request, #10556:
URL: https://github.com/apache/hudi/pull/10556
### Change Logs
hoodie.write.handle.missing.cols.with.lossless.type.promotion does not work with HoodieIncrSource unless meta cols are dropped
### Impact
allow hoodie.write.handle.missing.cols.with.lossless.type.promotion to be used with HoodieIncrSource even when meta cols aren't dropped
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1907080593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd66a2b6c21e32cc340e3a813acf826dc83b3547 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #10556:
URL: https://github.com/apache/hudi/pull/10556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1907072786
or lets add old one as alias to new one in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1907072458
hey @jonvex : guess we missed to cherry pick the config fix patch into 0.14.1
can you put up a patch to revert the naming change. we can keep it as old name only (hoodie.write.set.null.for.missing.columns)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1909242727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
}, {
"hash" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22151",
"triggerID" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1909099807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
}, {
"hash" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd66a2b6c21e32cc340e3a813acf826dc83b3547 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131)
* 6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1906791845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd66a2b6c21e32cc340e3a813acf826dc83b3547 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10556:
URL: https://github.com/apache/hudi/pull/10556#discussion_r1465590724
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -609,6 +610,7 @@ static HoodieDeltaStreamer.Config makeConfigForHudiIncrSrc(String srcBasePath, S
cfg.schemaProviderClassName = schemaProviderClassName;
}
List<String> cfgs = new ArrayList<>();
+ cfgs.add(HANDLE_MISSING_COLUMNS_WITH_LOSSLESS_TYPE_PROMOTIONS.key() + "=true");
Review Comment:
yes. Without the change the stream sync line 664 it fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1909106935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
}, {
"hash" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22151",
"triggerID" : "6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd66a2b6c21e32cc340e3a813acf826dc83b3547 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131)
* 6fb0ee2e5f0edcdf7657269973eb0968d0d7b0fa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "rmahindra123 (via GitHub)" <gi...@apache.org>.
rmahindra123 commented on code in PR #10556:
URL: https://github.com/apache/hudi/pull/10556#discussion_r1464148519
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -609,6 +610,7 @@ static HoodieDeltaStreamer.Config makeConfigForHudiIncrSrc(String srcBasePath, S
cfg.schemaProviderClassName = schemaProviderClassName;
}
List<String> cfgs = new ArrayList<>();
+ cfgs.add(HANDLE_MISSING_COLUMNS_WITH_LOSSLESS_TYPE_PROMOTIONS.key() + "=true");
Review Comment:
was the original issue reproducible with this config change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7327] remove meta cols from incoming schema in stream sync [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10556:
URL: https://github.com/apache/hudi/pull/10556#issuecomment-1906803120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131",
"triggerID" : "fd66a2b6c21e32cc340e3a813acf826dc83b3547",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd66a2b6c21e32cc340e3a813acf826dc83b3547 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22131)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org