You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/10 09:31:37 UTC
[GitHub] [hudi] fengjian428 opened a new pull request, #6355: add auto col cast in MergeIntoHoodieTableCommand
fengjian428 opened a new pull request, #6355:
URL: https://github.com/apache/hudi/pull/6355
### Change Logs
https://github.com/apache/hudi/issues/6354
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1260377382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766)
* 4cec796f80cd5979d893f792fd108a26d1ea614b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258937184
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262522730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
}, {
"hash" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833)
* 1323a940de5a28b75dcec8c422916032d6740fe3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900)
* 51fe330035a595e4d65cdf58554077ed0916fd25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1260376680
> @fengjian428 let's also add a test for this issue
at what level? I feel it looks redundant if create a test only to check the payload class, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258901721
> @fengjian428 please attach corresponding JIRA, and update PR description
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1210455809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ad4c06fb5eedab0725589e1958284301e751296 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1259016639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258771758
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262804005
CI is green:
<img width="1648" alt="Screen Shot 2022-09-29 at 1 52 39 PM" src="https://user-images.githubusercontent.com/428277/193139753-763ed18d-ee41-4e29-9eab-850c05f99912.png">
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=11905&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262450759
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833)
* 1323a940de5a28b75dcec8c422916032d6740fe3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262529064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
}, {
"hash" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1323a940de5a28b75dcec8c422916032d6740fe3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900)
* 51fe330035a595e4d65cdf58554077ed0916fd25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1261162597
@fengjian428 we should add additional case for Merge Into statement tests, where we specify non-default record-payload and make sure that it still works as expected. This is a regression, so we'd add a test for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1260625486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766)
* 4cec796f80cd5979d893f792fd108a26d1ea614b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823)
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1261172635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262834668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
}, {
"hash" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11905",
"triggerID" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51fe330035a595e4d65cdf58554077ed0916fd25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11905)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6355:
URL: https://github.com/apache/hudi/pull/6355#discussion_r980353314
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -150,15 +151,35 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// Create the write parameters
val parameters = buildMergeIntoConfig(hoodieCatalogTable)
+ val expectedQueryColumns = hoodieCatalogTable.tableSchemaWithoutMetaFields
+
+ val coercedQueryOutput = coerceQueryOutputColumns(StructType(expectedQueryColumns), sourceDF.logicalPlan, hoodieCatalogTable, conf)
Review Comment:
understood Thanks @alexeykudinkin
essentially， I realized that this issue was caused by we must use ExpressionPayload in MergeInto, but for now, write.payload.class can be overwritten by the user, so I changed the logic forced to use ExpressionPayload in MergeInto. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258638641
@fengjian428 please attach corresponding JIRA, and update PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262516567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833)
* 1323a940de5a28b75dcec8c422916032d6740fe3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1210601327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ad4c06fb5eedab0725589e1958284301e751296 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1251512909
@xushiyan i don't think we will be able to land this as is, as it might heavily restrict usability of the `MERGE INTO`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262534555
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
}, {
"hash" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900",
"triggerID" : "1323a940de5a28b75dcec8c422916032d6740fe3",
"triggerType" : "PUSH"
}, {
"hash" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11905",
"triggerID" : "51fe330035a595e4d65cdf58554077ed0916fd25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1323a940de5a28b75dcec8c422916032d6740fe3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11900)
* 51fe330035a595e4d65cdf58554077ed0916fd25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11905)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258468508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ad4c06fb5eedab0725589e1958284301e751296 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722)
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1210461396
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ad4c06fb5eedab0725589e1958284301e751296 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1250125289
@alexeykudinkin can you recommend a plan: land this alone or include the fix in https://github.com/apache/hudi/pull/6361 ? both are for https://github.com/apache/hudi/issues/6354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258961381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1260409157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766)
* 4cec796f80cd5979d893f792fd108a26d1ea614b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1262445510
> @fengjian428 we should add additional case for Merge Into statement tests, where we specify non-default record-payload and make sure that it still works as expected. This is a regression, so we'd add a test for it
ok, add a ut to cover https://github.com/apache/hudi/issues/6354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6355:
URL: https://github.com/apache/hudi/pull/6355#discussion_r974659357
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -150,15 +151,35 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
// Create the write parameters
val parameters = buildMergeIntoConfig(hoodieCatalogTable)
+ val expectedQueryColumns = hoodieCatalogTable.tableSchemaWithoutMetaFields
+
+ val coercedQueryOutput = coerceQueryOutputColumns(StructType(expectedQueryColumns), sourceDF.logicalPlan, hoodieCatalogTable, conf)
Review Comment:
We can't do that unfortunately: `MERGE INTO` semantic does allow you to merge arbitrary input that might be drastically different from your target table, controlling what goes where w/ matched/non-matched statements (`t.id = s._other_id`, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #6355:
URL: https://github.com/apache/hudi/pull/6355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6355:
URL: https://github.com/apache/hudi/pull/6355#discussion_r980509048
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -508,6 +507,6 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
SqlKeyGenerator.PARTITION_SCHEMA -> partitionSchema.toDDL
)
.filter { case (_, v) => v != null }
- }
+ } + (PAYLOAD_CLASS_NAME.key -> classOf[ExpressionPayload].getCanonicalName)
Review Comment:
No need to append it here -- you can prepend to `catalogProperties` above.
Let's also add a comment explaining why this is put in here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: [HUDI-4925] Should Force to use ExpressionPayload in MergeIntoTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1260632862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11748",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11766",
"triggerID" : "1258937184",
"triggerType" : "MANUAL"
}, {
"hash" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823",
"triggerID" : "4cec796f80cd5979d893f792fd108a26d1ea614b",
"triggerType" : "PUSH"
}, {
"hash" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833",
"triggerID" : "90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4cec796f80cd5979d893f792fd108a26d1ea614b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11823)
* 90dd4ca5753b5bf1ce434bc1ab3a05b5381a9aa0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6355: add auto col cast in MergeIntoHoodieTableCommand
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6355:
URL: https://github.com/apache/hudi/pull/6355#issuecomment-1258463055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ad4c06fb5eedab0725589e1958284301e751296",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722",
"triggerID" : "9ad4c06fb5eedab0725589e1958284301e751296",
"triggerType" : "PUSH"
}, {
"hash" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a774eec1b4293fadf9b99638c96a0fb3bedc8d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ad4c06fb5eedab0725589e1958284301e751296 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10722)
* 4a774eec1b4293fadf9b99638c96a0fb3bedc8d1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org