You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/26 04:24:12 UTC
[GitHub] [hudi] XuQianJin-Stars opened a new pull request, #6216: [HUDI-4475] fix create table with not exists hoodie properties file
XuQianJin-Stars opened a new pull request, #6216:
URL: https://github.com/apache/hudi/pull/6216
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1194987238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18d96c0ec6b56767f37faf8ca988b1bc3243ec46 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1205950765
we added support to update hoodie.properties in a live env. mainly to update some table propertlies like metadata related props (list of partitions in metadata table). So, here is how upgrade works so that its fault tolerant and recoverable.
orig.hoodie.properties
Step1:
take back up.
cp orig.hoodie.properties backup.hoodie.properties.
Step2:
delete orig.hoodie.properties
Step3:
create new hoodie.properties in memory w/ any new properties required.
create orig.hoodie.properties.
Step4:
delete backup.hoodie.properties.
b/w step2 and step3, readers will read backup.hoodie.properties.
Above is designed such that, if there is a crash at any point, we are safe and restarting the pipeline would suffice.
ref: https://github.com/apache/hudi/blob/a75cc02273ae87c383ae1ed46f95006c366f70fc/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
YannByron commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1195032436
What kind of operations would need to cause this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1226338378
> > > What kind of operations would need to cause this case?
> >
> >
> > Because now the production environment finds that re-deleting the table (non-purge) and some other abnormal scenarios of compcation will cause the hudi properties file to be deleted, and then the re-creation of the table will fail.
>
> I think we need to figure out the root cause why compaction deletes `hoodie.properties`. Also can we add ut for deleting the `hoodie.properties` manually and see whether the writing works well?
@leesf @XuQianJin-Stars I agree that we should fix the root cause. This patch is more like treating the symptom. @XuQianJin-Stars let's close this and try reproduce the problem you had? if any corner case where properties file was unexpectedly deleted, it should be fixed most likely around the code in https://github.com/apache/hudi/blob/a75cc02273ae87c383ae1ed46f95006c366f70fc/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L344 as mentioned by @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1195066926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10345",
"triggerID" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18d96c0ec6b56767f37faf8ca988b1bc3243ec46 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
leesf commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1195259725
> > What kind of operations would need to cause this case?
>
> Because now the production environment finds that re-deleting the table (non-purge) and some other abnormal scenarios of compcation will cause the hudi properties file to be deleted, and then the re-creation of the table will fail.
I think we need to figure out the root cause why compaction deletes `hoodie.properties`. Also can we add ut for deleting the `hoodie.properties` manually and see whether the writing works well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1194989868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10345",
"triggerID" : "18d96c0ec6b56767f37faf8ca988b1bc3243ec46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18d96c0ec6b56767f37faf8ca988b1bc3243ec46 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars closed pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars closed pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
URL: https://github.com/apache/hudi/pull/6216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6216: [HUDI-4475] fix create table with not exists hoodie properties file
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6216:
URL: https://github.com/apache/hudi/pull/6216#issuecomment-1195122063
> What kind of operations would need to cause this case?
Because now the production environment finds that re-deleting the table (non-purge) and some other abnormal scenarios of compcation will cause the hudi properties file to be deleted, and then the re-creation of the table will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org