You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/05/04 23:20:26 UTC
[GitHub] [hudi] jonvex opened a new pull request, #8638: added new exception types
jonvex opened a new pull request, #8638:
URL: https://github.com/apache/hudi/pull/8638
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #8638:
URL: https://github.com/apache/hudi/pull/8638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1548214543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840)
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6898285dd6ddba725ae33b73aa92afa02beb98a7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1557399470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 2da0fa482136c6179fd14df82843f32f4a0877e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218)
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1535634029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1554050386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* aa35b5562c16840b5ebf143009beac2c291de2c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1559868609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267)
* 2b0d0627582948948f230f4eeaa07e9289827d7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1557323864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 2da0fa482136c6179fd14df82843f32f4a0877e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218)
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1553649953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6898285dd6ddba725ae33b73aa92afa02beb98a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090)
* aa35b5562c16840b5ebf143009beac2c291de2c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8638: added new exception types
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561759612
you can land it once CI is green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561935852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
}, {
"hash" : "38111848f9b39572083b87d785d2002b01d00d99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351",
"triggerID" : "38111848f9b39572083b87d785d2002b01d00d99",
"triggerType" : "PUSH"
}, {
"hash" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* b34d619d7162d5d70012f4a0b186f2f0f0810455 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312)
* 38111848f9b39572083b87d785d2002b01d00d99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351)
* a2ba4b9184beb226de03bdd0c5ff316d28725971 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1553638442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6898285dd6ddba725ae33b73aa92afa02beb98a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090)
* aa35b5562c16840b5ebf143009beac2c291de2c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1202697891
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/RowBasedSchemaProvider.java:
##########
@@ -44,7 +45,12 @@ public RowBasedSchemaProvider(StructType rowStruct) {
@Override
public Schema getSourceSchema() {
- return AvroConversionUtils.convertStructTypeToAvroSchema(rowStruct, HOODIE_RECORD_STRUCT_NAME,
- HOODIE_RECORD_NAMESPACE);
+ try {
+ return AvroConversionUtils.convertStructTypeToAvroSchema(rowStruct, HOODIE_RECORD_STRUCT_NAME,
+ HOODIE_RECORD_NAMESPACE);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to convert struct type to avro schema", e);
Review Comment:
Good catch. I actually do throw it in there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1555244224
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 2da0fa482136c6179fd14df82843f32f4a0877e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1192524362
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/exception/HoodieDeltaStreamerMetaSyncException.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.exception;
+
+public class HoodieDeltaStreamerMetaSyncException extends HoodieDeltaStreamerException {
Review Comment:
For these new exception types, can we capture some context like which meta sync failed?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/exception/HoodieDeltaStreamerSchemaCompatibilityException.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.exception;
+
+public class HoodieDeltaStreamerSchemaCompatibilityException extends HoodieDeltaStreamerException {
Review Comment:
Similarly is it possible to catch a before/after schema here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1194396609
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/SqlQueryBasedTransformer.java:
##########
@@ -47,17 +49,21 @@ public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Datas
TypedProperties properties) {
String transformerSQL = properties.getString(SqlTransformerConfig.TRANSFORMER_SQL.key());
if (null == transformerSQL) {
- throw new IllegalArgumentException("Missing configuration : (" + SqlTransformerConfig.TRANSFORMER_SQL.key() + ")");
+ throw new HoodieDeltaStreamerTransformPlanException("Missing configuration : (" + SqlTransformerConfig.TRANSFORMER_SQL.key() + ")");
}
- // tmp table name doesn't like dashes
- String tmpTable = TMP_TABLE.concat(UUID.randomUUID().toString().replace("-", "_"));
- LOG.info("Registering tmp table : " + tmpTable);
- rowDataset.createOrReplaceTempView(tmpTable);
- String sqlStr = transformerSQL.replaceAll(SRC_PATTERN, tmpTable);
- LOG.debug("SQL Query for transformation : (" + sqlStr + ")");
- Dataset<Row> transformed = sparkSession.sql(sqlStr);
- sparkSession.catalog().dropTempView(tmpTable);
- return transformed;
+ try {
+ // tmp table name doesn't like dashes
+ String tmpTable = TMP_TABLE.concat(UUID.randomUUID().toString().replace("-", "_"));
+ LOG.info("Registering tmp table : " + tmpTable);
+ rowDataset.createOrReplaceTempView(tmpTable);
+ String sqlStr = transformerSQL.replaceAll(SRC_PATTERN, tmpTable);
+ LOG.debug("SQL Query for transformation : (" + sqlStr + ")");
+ Dataset<Row> transformed = sparkSession.sql(sqlStr);
+ sparkSession.catalog().dropTempView(tmpTable);
+ return transformed;
+ } catch (Exception e) {
+ throw new HoodieDeltaStreamerTransformExecutionException("Failed to apply sql query based transformer", e);
Review Comment:
Will this catch execution exceptions or just exceptions related to the spark plan that is generated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1202691156
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -138,18 +139,26 @@ object AvroConversionUtils {
def convertStructTypeToAvroSchema(structType: DataType,
structName: String,
recordNamespace: String): Schema = {
- val schemaConverters = sparkAdapter.getAvroSchemaConverters
- val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
- getAvroSchemaWithDefaults(avroSchema, structType)
+ try {
+ val schemaConverters = sparkAdapter.getAvroSchemaConverters
+ val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
Review Comment:
I don't think we can. It looks like it's a spark internal class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561797925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
}, {
"hash" : "38111848f9b39572083b87d785d2002b01d00d99",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38111848f9b39572083b87d785d2002b01d00d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* b34d619d7162d5d70012f4a0b186f2f0f0810455 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312)
* 38111848f9b39572083b87d785d2002b01d00d99 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1546706518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840)
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1194295333
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/exception/HoodieDeltaStreamerSchemaCompatibilityException.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.exception;
+
+public class HoodieDeltaStreamerSchemaCompatibilityException extends HoodieDeltaStreamerException {
Review Comment:
I think that depends on where it is thrown. I think there are 6 places it is thrown. Could you please point out which ones and what they should provide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1560433831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* b34d619d7162d5d70012f4a0b186f2f0f0810455 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1201011129
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -138,18 +139,26 @@ object AvroConversionUtils {
def convertStructTypeToAvroSchema(structType: DataType,
structName: String,
recordNamespace: String): Schema = {
- val schemaConverters = sparkAdapter.getAvroSchemaConverters
- val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
- getAvroSchemaWithDefaults(avroSchema, structType)
+ try {
+ val schemaConverters = sparkAdapter.getAvroSchemaConverters
+ val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
Review Comment:
Should the `schemaConverters.toAvroType` throw the HoodieSchemaException instead of this util class?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/RowBasedSchemaProvider.java:
##########
@@ -44,7 +45,12 @@ public RowBasedSchemaProvider(StructType rowStruct) {
@Override
public Schema getSourceSchema() {
- return AvroConversionUtils.convertStructTypeToAvroSchema(rowStruct, HOODIE_RECORD_STRUCT_NAME,
- HOODIE_RECORD_NAMESPACE);
+ try {
+ return AvroConversionUtils.convertStructTypeToAvroSchema(rowStruct, HOODIE_RECORD_STRUCT_NAME,
+ HOODIE_RECORD_NAMESPACE);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to convert struct type to avro schema", e);
Review Comment:
Should we move this throw into the AvroConversionUtils? I see other methods in there throw the exception themselves.
##########
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetaSyncException.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+public class HoodieMetaSyncException extends HoodieException {
Review Comment:
Can we include the list of syncs involved so we can easily handle different sync failures with custom logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1202703409
##########
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetaSyncException.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+public class HoodieMetaSyncException extends HoodieException {
Review Comment:
So the message will contain all the other syncs. For custom logic, I don't think it would be handled using the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1559936091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267)
* 2b0d0627582948948f230f4eeaa07e9289827d7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305)
* b34d619d7162d5d70012f4a0b186f2f0f0810455 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1559946073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 2b0d0627582948948f230f4eeaa07e9289827d7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305)
* b34d619d7162d5d70012f4a0b186f2f0f0810455 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561806977
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
}, {
"hash" : "38111848f9b39572083b87d785d2002b01d00d99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351",
"triggerID" : "38111848f9b39572083b87d785d2002b01d00d99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* b34d619d7162d5d70012f4a0b186f2f0f0810455 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312)
* 38111848f9b39572083b87d785d2002b01d00d99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1562503283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
}, {
"hash" : "38111848f9b39572083b87d785d2002b01d00d99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351",
"triggerID" : "38111848f9b39572083b87d785d2002b01d00d99",
"triggerType" : "PUSH"
}, {
"hash" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17354",
"triggerID" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* a2ba4b9184beb226de03bdd0c5ff316d28725971 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1554786752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* aa35b5562c16840b5ebf143009beac2c291de2c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196)
* 2da0fa482136c6179fd14df82843f32f4a0877e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1554797583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* aa35b5562c16840b5ebf143009beac2c291de2c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196)
* 2da0fa482136c6179fd14df82843f32f4a0877e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1202703409
##########
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetaSyncException.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+public class HoodieMetaSyncException extends HoodieException {
Review Comment:
So the message will contain all the other syncs. They could be parsed out but I guess I could add them in a list at the beginning of the exception to make parsing easier. For custom logic, I don't think it would be handled using the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1202720999
##########
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetaSyncException.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+public class HoodieMetaSyncException extends HoodieException {
Review Comment:
You can make the class have a field and then a getter for that field that people can use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1203453231
##########
hudi-common/src/main/java/org/apache/hudi/exception/HoodieMetaSyncException.java:
##########
@@ -18,12 +18,18 @@
package org.apache.hudi.exception;
+import java.util.Map;
+
public class HoodieMetaSyncException extends HoodieException {
+
+ public Map<String,HoodieException> failedMetaSyncs;
Review Comment:
make this private and add a getter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #8638: added new exception types
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1204601709
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java:
##########
@@ -193,8 +201,8 @@ public Schema getTargetSchema() {
String targetRegistryUrl = config.getString(HoodieSchemaProviderConfig.TARGET_SCHEMA_REGISTRY_URL.key(), registryUrl);
try {
return parseSchemaFromRegistry(targetRegistryUrl);
- } catch (IOException ioe) {
- throw new HoodieIOException("Error reading target schema from registry :" + registryUrl, ioe);
+ } catch (Exception e) {
Review Comment:
yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: [HUDI-6261] Make deltastreamer exceptions more descriptive
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561944329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17305",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
}, {
"hash" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17312",
"triggerID" : "b34d619d7162d5d70012f4a0b186f2f0f0810455",
"triggerType" : "PUSH"
}, {
"hash" : "38111848f9b39572083b87d785d2002b01d00d99",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351",
"triggerID" : "38111848f9b39572083b87d785d2002b01d00d99",
"triggerType" : "PUSH"
}, {
"hash" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17354",
"triggerID" : "a2ba4b9184beb226de03bdd0c5ff316d28725971",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 38111848f9b39572083b87d785d2002b01d00d99 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17351)
* a2ba4b9184beb226de03bdd0c5ff316d28725971 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1548756828
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6898285dd6ddba725ae33b73aa92afa02beb98a7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1545937630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840)
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #8638: added new exception types
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1194391041
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -1119,7 +1129,7 @@ private Schema getSchemaForWriteConfig(Schema targetSchema) {
}
return newWriteSchema;
} catch (Exception e) {
- throw new HoodieException("Failed to fetch schema from table ", e);
+ throw new HoodieDeltaStreamerSchemaFetchException("Failed to fetch schema from table ", e);
Review Comment:
nitpick: don't need a trailing space
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java:
##########
@@ -108,9 +109,13 @@ private void initJsonConvertor() {
}
public GenericRecord fromJson(String json) {
- initSchema();
- initJsonConvertor();
- return jsonConverter.convert(json, schema);
+ try {
+ initSchema();
+ initJsonConvertor();
+ return jsonConverter.convert(json, schema);
+ } catch (Exception e) {
+ throw new HoodieDeltaStreamerSchemaCompatibilityException("Failed to convert schema from json to avro", e);
Review Comment:
We want to distinguish between parsing/conversion style errors and compatibility errors. Compatibility is more like something violates the supported schema evolution [here](https://hudi.apache.org/docs/schema_evolution#out-of-the-box-schema-evolution)
For all of the errors in this class, they fall into a parsing/conversion style error
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -679,8 +683,12 @@ private Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> fetchFromSourc
}
private JavaRDD<GenericRecord> getTransformedRDD(Dataset<Row> rowDataset, boolean reconcileSchema, Schema readerSchema) {
- return HoodieSparkUtils.createRdd(rowDataset, HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, reconcileSchema,
- Option.ofNullable(readerSchema)).toJavaRDD();
+ try {
+ return HoodieSparkUtils.createRdd(rowDataset, HOODIE_RECORD_STRUCT_NAME, HOODIE_RECORD_NAMESPACE, reconcileSchema,
+ Option.ofNullable(readerSchema)).toJavaRDD();
+ } catch (Exception e) {
+ throw new HoodieDeltaStreamerSchemaCompatibilityException("Failed to get transformed RDD", e);
Review Comment:
Do we know that this will always be a schema issue?
What is the readSchema at this point? We could include the context of this dataset's schema (possibly converted to avro) and the readerSchema
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -970,10 +979,11 @@ public void runMetaSync() {
} catch (HoodieException e) {
LOG.info("SyncTool class " + impl.trim() + " failed with exception", e);
metaSyncExceptions.add(e);
+ implsFailed.add(impl.trim());
}
}
if (!metaSyncExceptions.isEmpty()) {
- throw SyncUtilHelpers.getExceptionFromList(metaSyncExceptions);
+ throw new HoodieDeltaStreamerMetaSyncException("Meta sync failure for " + String.join(",", implsFailed), SyncUtilHelpers.getExceptionFromList(metaSyncExceptions));
Review Comment:
It may be easier to work with the exception if we have it take in the list of `implsfailed` and then construct the message in the call to `super`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1535550076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1535544415
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1559857438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
}, {
"hash" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b0d0627582948948f230f4eeaa07e9289827d7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267)
* 2b0d0627582948948f230f4eeaa07e9289827d7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8638: added new exception types
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1557893116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16840",
"triggerID" : "cffdd5242a58a50e4ce11a5d098d4ab7f4da1ec8",
"triggerType" : "PUSH"
}, {
"hash" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc",
"triggerType" : "PUSH"
}, {
"hash" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "333d9faa53e71ba535a7cb8c60ce8b350a33452c",
"triggerType" : "PUSH"
}, {
"hash" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17090",
"triggerID" : "6898285dd6ddba725ae33b73aa92afa02beb98a7",
"triggerType" : "PUSH"
}, {
"hash" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17196",
"triggerID" : "aa35b5562c16840b5ebf143009beac2c291de2c9",
"triggerType" : "PUSH"
}, {
"hash" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17218",
"triggerID" : "2da0fa482136c6179fd14df82843f32f4a0877e3",
"triggerType" : "PUSH"
}, {
"hash" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267",
"triggerID" : "6cbafd0d08e12fc4e77a9f0058fe24b23e352a69",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c8cf2d86b1be30d3215b3b6e89b8bda33a1fe5dc UNKNOWN
* 333d9faa53e71ba535a7cb8c60ce8b350a33452c UNKNOWN
* 6cbafd0d08e12fc4e77a9f0058fe24b23e352a69 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17267)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8638: added new exception types
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8638:
URL: https://github.com/apache/hudi/pull/8638#discussion_r1203336055
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -138,18 +139,26 @@ object AvroConversionUtils {
def convertStructTypeToAvroSchema(structType: DataType,
structName: String,
recordNamespace: String): Schema = {
- val schemaConverters = sparkAdapter.getAvroSchemaConverters
- val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
- getAvroSchemaWithDefaults(avroSchema, structType)
+ try {
+ val schemaConverters = sparkAdapter.getAvroSchemaConverters
+ val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
+ getAvroSchemaWithDefaults(avroSchema, structType)
+ } catch {
+ case e: Exception => throw new HoodieSchemaException("Failed to convert struct type to avro schema", e)
Review Comment:
do you think we should log the structType here?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -138,18 +139,26 @@ object AvroConversionUtils {
def convertStructTypeToAvroSchema(structType: DataType,
structName: String,
recordNamespace: String): Schema = {
- val schemaConverters = sparkAdapter.getAvroSchemaConverters
- val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
- getAvroSchemaWithDefaults(avroSchema, structType)
+ try {
+ val schemaConverters = sparkAdapter.getAvroSchemaConverters
+ val avroSchema = schemaConverters.toAvroType(structType, nullable = false, structName, recordNamespace)
+ getAvroSchemaWithDefaults(avroSchema, structType)
+ } catch {
+ case e: Exception => throw new HoodieSchemaException("Failed to convert struct type to avro schema", e)
+ }
}
/**
* Converts Avro's [[Schema]] to Catalyst's [[StructType]]
*/
def convertAvroSchemaToStructType(avroSchema: Schema): StructType = {
- val schemaConverters = sparkAdapter.getAvroSchemaConverters
- schemaConverters.toSqlType(avroSchema) match {
- case (dataType, _) => dataType.asInstanceOf[StructType]
+ try {
+ val schemaConverters = sparkAdapter.getAvroSchemaConverters
+ schemaConverters.toSqlType(avroSchema) match {
+ case (dataType, _) => dataType.asInstanceOf[StructType]
+ }
+ } catch {
+ case e: Exception => throw new HoodieSchemaException("Failed to convert avro schema to struct type", e)
Review Comment:
same here. should we log the schema here?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/KafkaOffsetPostProcessor.java:
##########
@@ -59,13 +60,18 @@ public KafkaOffsetPostProcessor(TypedProperties props, JavaSparkContext jssc) {
@Override
public Schema processSchema(Schema schema) {
// this method adds kafka offset fields namely source offset, partition and timestamp to the schema of the batch.
- List<Schema.Field> fieldList = schema.getFields();
- List<Schema.Field> newFieldList = fieldList.stream()
- .map(f -> new Schema.Field(f.name(), f.schema(), f.doc(), f.defaultVal())).collect(Collectors.toList());
- newFieldList.add(new Schema.Field(KAFKA_SOURCE_OFFSET_COLUMN, Schema.create(Schema.Type.LONG), "offset column", 0));
- newFieldList.add(new Schema.Field(KAFKA_SOURCE_PARTITION_COLUMN, Schema.create(Schema.Type.INT), "partition column", 0));
- newFieldList.add(new Schema.Field(KAFKA_SOURCE_TIMESTAMP_COLUMN, Schema.create(Schema.Type.LONG), "timestamp column", 0));
- Schema newSchema = Schema.createRecord(schema.getName() + "_processed", schema.getDoc(), schema.getNamespace(), false, newFieldList);
- return newSchema;
+ try {
+ List<Schema.Field> fieldList = schema.getFields();
+ List<Schema.Field> newFieldList = fieldList.stream()
+ .map(f -> new Schema.Field(f.name(), f.schema(), f.doc(), f.defaultVal())).collect(Collectors.toList());
+ newFieldList.add(new Schema.Field(KAFKA_SOURCE_OFFSET_COLUMN, Schema.create(Schema.Type.LONG), "offset column", 0));
+ newFieldList.add(new Schema.Field(KAFKA_SOURCE_PARTITION_COLUMN, Schema.create(Schema.Type.INT), "partition column", 0));
+ newFieldList.add(new Schema.Field(KAFKA_SOURCE_TIMESTAMP_COLUMN, Schema.create(Schema.Type.LONG), "timestamp column", 0));
+ Schema newSchema = Schema.createRecord(schema.getName() + "_processed", schema.getDoc(), schema.getNamespace(), false, newFieldList);
+ return newSchema;
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Kafka offset post processor failed", e);
Review Comment:
can we log the schema
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java:
##########
@@ -89,12 +90,16 @@ public interface SchemaConverter {
String convert(String schema) throws IOException;
}
- public Schema parseSchemaFromRegistry(String registryUrl) throws IOException {
+ public Schema parseSchemaFromRegistry(String registryUrl) {
String schema = fetchSchemaFromRegistry(registryUrl);
- SchemaConverter converter = config.containsKey(HoodieSchemaProviderConfig.SCHEMA_CONVERTER.key())
- ? ReflectionUtils.loadClass(config.getString(HoodieSchemaProviderConfig.SCHEMA_CONVERTER.key()))
- : s -> s;
- return new Schema.Parser().parse(converter.convert(schema));
+ try {
+ SchemaConverter converter = config.containsKey(HoodieSchemaProviderConfig.SCHEMA_CONVERTER.key())
+ ? ReflectionUtils.loadClass(config.getString(HoodieSchemaProviderConfig.SCHEMA_CONVERTER.key()))
+ : s -> s;
+ return new Schema.Parser().parse(converter.convert(schema));
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to parse schema from registry", e);
Review Comment:
can we log the schema
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java:
##########
@@ -108,9 +109,13 @@ private void initJsonConvertor() {
}
public GenericRecord fromJson(String json) {
- initSchema();
- initJsonConvertor();
- return jsonConverter.convert(json, schema);
+ try {
+ initSchema();
+ initJsonConvertor();
+ return jsonConverter.convert(json, schema);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to convert schema from json to avro", e);
Review Comment:
if schema is not null, can we log schema
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/ProtoClassBasedSchemaProvider.java:
##########
@@ -84,8 +84,13 @@ public ProtoClassBasedSchemaProvider(TypedProperties props, JavaSparkContext jss
@Override
public Schema getSourceSchema() {
if (schema == null) {
- Schema.Parser parser = new Schema.Parser();
- schema = parser.parse(schemaString);
+ try {
+ Schema.Parser parser = new Schema.Parser();
+ schema = parser.parse(schemaString);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to parse schema", e);
Review Comment:
can we log schemaString
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java:
##########
@@ -124,18 +129,31 @@ public Either<GenericRecord,String> fromJsonWithError(String json) {
}
public Schema getSchema() {
- return new Schema.Parser().parse(schemaStr);
+ try {
+ return new Schema.Parser().parse(schemaStr);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to parse json schema", e);
Review Comment:
lets log schemaStr
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/exception/HoodieTransformExecutionException.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.exception;
Review Comment:
can you add java docs on whats the diff b/w this and HoodieTransformException ?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java:
##########
@@ -124,18 +129,31 @@ public Either<GenericRecord,String> fromJsonWithError(String json) {
}
public Schema getSchema() {
- return new Schema.Parser().parse(schemaStr);
+ try {
+ return new Schema.Parser().parse(schemaStr);
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to parse json schema", e);
+ }
}
public GenericRecord fromAvroBinary(byte[] avroBinary) {
- initSchema();
- initInjection();
- return recordInjection.invert(avroBinary).get();
+ try {
+ initSchema();
+ initInjection();
+ return recordInjection.invert(avroBinary).get();
+ } catch (Exception e) {
+ throw new HoodieSchemaException("Failed to get avro schema from avro binary", e);
Review Comment:
same here
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java:
##########
@@ -193,8 +201,8 @@ public Schema getTargetSchema() {
String targetRegistryUrl = config.getString(HoodieSchemaProviderConfig.TARGET_SCHEMA_REGISTRY_URL.key(), registryUrl);
try {
return parseSchemaFromRegistry(targetRegistryUrl);
- } catch (IOException ioe) {
- throw new HoodieIOException("Error reading target schema from registry :" + registryUrl, ioe);
+ } catch (Exception e) {
Review Comment:
shouldn't we log targetRegistryUrl instead of registryUrl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8638: added new exception types
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8638:
URL: https://github.com/apache/hudi/pull/8638#issuecomment-1561759372
Please prefix hudi jira. and update the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org