You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Rajeev-01 (via GitHub)" <gi...@apache.org> on 2025/06/22 17:50:54 UTC
[PR] Make the creation of properties file atomic during modification [hudi]
Rajeev-01 opened a new pull request, #13473:
URL: https://github.com/apache/hudi/pull/13473
### Change Logs
- Making the creation of properties file atomic during the modification.
- This is required since if the create calls fails with Unexpected exception which will create the empty file.
- This empty file creation and next modify call will delete the backup file which has correct configs and both backup file and actual file becomes empty and this PR addresses this issue.
### Impact
- This will make sure that either properties file or properties.backup file will be present with the correct properties.
- Even if we face the above mentioned issue this time the temp file will get affected.
- The left over temp file will get overwritten by the next modify call and make sure that properties won't be lost.
### Testing
- Existing UT have to pass which is already testing the multiple writers trying to modify the properties.
### Documentation Update
none
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3239034636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
}, {
"hash" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 725fe0b9b12286a2953ab50818478fdd33c71efd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869)
* a65cb91ef3600eb7fcc81e7978ba8882528c29a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2312451673
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Anyways the prechecks added will take care that no corrupted properties will be loaded.
Cool let me close the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193563731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865)
* 83ee9f980434afd65f7bd626ff123e52855e0799 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868)
* 725fe0b9b12286a2953ab50818478fdd33c71efd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288200767
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
> It looks like what is really problematic is the #recoverIfNeeded, it just check the file existence of the cfgPath but not the checksum, which could incur empty files like you mentioned.
Yeah exactly. I can address this is next PR
> I did't see the issues, just try to understand why this is needed, we already have backup file and checksum validation for cfg file, when the file is empty, the checksum would be invalid, and the reader would just read from backup file.
Just think if this issue happened twice in a row in a different manner.
first actual pros will get empty
Next one since actual file is present and we will drop the backup file which contains actual props
and while populating the backup again we will end up in 0 bytes for the back up as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2187943064
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah, but due to these exceptions we will hit during create itself and since we are not catching them, we won't reach the check sum comparision check itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2160650517
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
We have `HoodieStorage.createImmutableFileInPath` that can be utilized to ensure file creation atomicity if that is your purpose. BTW, if the file creation is atomic, then the `backupCfgPath` related logic can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193559495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865)
* 83ee9f980434afd65f7bd626ff123e52855e0799 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288265399
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
And also this needs to check if the backup is valid or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2188902814
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
> 3. SInce properties file is present but empty, we won't recover from backup file and we will delete the backup file which contains the actual properties.
Can we just fix 3 by checking whether the existing properties file contains a checksum there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193363400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eee2065f1c05687b82a611f63de189438553533c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864)
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193362479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eee2065f1c05687b82a611f63de189438553533c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288153824
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
I did't see the issues, just try to understand why this is needed, we already have backup file and checksum validation for cfg file, when the file is empty, the checksum would be invalid, and the reader would just read from backup file.
It looks like what is really problematic is the `#recoverIfNeeded`, it just check the file existence of the `cfgPath` but not the checksum, which could incur empty files like you mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2160690240
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
1. `createImmutableFileInPath` will create the temp files with UUID which can leave the stray files if we use it for properties modification. If we use the `.temp` suffix even if the file is present we will overwrite the existing file.
2. Right now during modification the properties file is being generated in this way create -> write
But if the create call is being failed due to TCP connection failure (in HDFS due to connection failure between NN and clinet) and we will end hitting exception and it is ending up in NN lease recovery process will end up in closing the File with size 0.
3. So this temp file write and rename will ensure that properties file will be created with valid properties or it won't be present.
4. The backup logic needs to be present to serve readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2160690240
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Right now during modification the properties file is being generated in this way create -> write
But if the create call is being failed due to TCP connection failure (in HDFS due to connection failure between NN and clinet) and we will end hitting exception and it is ending up in NN lease recovery process will end up in closing the File with size 0. So this temp file write and rename will ensure that properties file will be created with valid properties or it won't be present.
The backup logic needs to be present to serve readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193361337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca048517ee0d0e62b57bfc47a2c5e2efca99439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413)
* eee2065f1c05687b82a611f63de189438553533c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193560781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865)
* 83ee9f980434afd65f7bd626ff123e52855e0799 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 closed pull request #13473: Make the creation of properties file atomic during modification
URL: https://github.com/apache/hudi/pull/13473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994347016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7df2acc0135dc8d64d8e542d435ee363afedb5f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994932154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7df2acc0135dc8d64d8e542d435ee363afedb5f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400)
* ca048517ee0d0e62b57bfc47a2c5e2efca99439e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994929203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7df2acc0135dc8d64d8e542d435ee363afedb5f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400)
* ca048517ee0d0e62b57bfc47a2c5e2efca99439e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2188807651
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Then the reader would firstly try to revover from the backup file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288224453
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Here is the fix for it:
```java
public static void recoverIfNeeded(HoodieStorage storage, StoragePath cfgPath,
StoragePath backupCfgPath) throws IOException {
boolean needCopy = false;
if (!storage.exists(cfgPath)) {
needCopy = true;
} else {
Properties props = new Properties();
try (InputStream in = storage.open(cfgPath)) {
props.load(in);
if (!props.containsKey(TABLE_CHECKSUM.key()) || !HoodieTableConfig.validateChecksum(props)) {
storage.deleteFile(cfgPath);
// the cfg file is invalid
needCopy = true;
}
}
}
if (needCopy) {
// copy over from backup
try (InputStream in = storage.open(backupCfgPath);
OutputStream out = storage.create(cfgPath, false)) {
FileIOUtils.copy(in, out);
}
}
// regardless, we don't need the backup anymore.
storage.deleteFile(backupCfgPath);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2190737453
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah, I also thought this will fix this issue, but here aslo we are going to populate it by creating and then writing. This will again the same issue we are trying to solve but for the backup file. So it's better to fix this issue for original one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2191393119
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
I didn't get your point, can you elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193392741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288259919
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah this will works and also we need to add the similar one while populating backup from original one as well and It's like a protection check. Can I update this PR or create new one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994353640
@vinothchandar can you review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2160690240
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
1. `createImmutableFileInPath` will create the temp files with UUID which can leave the stray files if we use it for properties modification. If we use the `.temp` suffix even if the file is present we will overwrite the existing file.
2. Right now during modification the properties file is being generated in this way create -> write
But if the create call is being failed due to TCP connection failure (in HDFS due to connection failure between NN and client) and we will end hitting exception and it is ending up in NN lease recovery process will end up in closing the File with size 0.
3. So this temp file write and rename will ensure that properties file will be created with valid properties or it won't be present.
4. The backup logic needs to be present to serve readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2995059003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca048517ee0d0e62b57bfc47a2c5e2efca99439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2187355454
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
@danny0405 addrssed the comments
can you take a look into this again? 😅
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
@danny0405 addressed the comments
can you take a look into this again? 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2188816891
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
1. The first instant when the properties file became empty, the backup file serve the readers.
2. When the next update came for properties file, we are checking if properties file is not present then recover from the backup file.
3. SInce properties file is present but empty, we won't recover from backup file and we will delete the backup file which contains the actual properties.
4. This ends up in both properties and backup become empty.
5. This temp file logic will make sure that the original will be present if it has been populated correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2312188229
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
I think this is unnecessary now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3239037753
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
}, {
"hash" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8281",
"triggerID" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 725fe0b9b12286a2953ab50818478fdd33c71efd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869)
* a65cb91ef3600eb7fcc81e7978ba8882528c29a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3239134432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
}, {
"hash" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8281",
"triggerID" : "a65cb91ef3600eb7fcc81e7978ba8882528c29a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a65cb91ef3600eb7fcc81e7978ba8882528c29a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2187896095
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
> Right now during modification the properties file is being generated in this way create -> write
But if the create call is being failed due to TCP connection failure (in HDFS due to connection failure between NN and client) and we will end hitting exception and it is ending up in NN lease recovery process will end up in closing the File with size 0
That is why the checksum palys a role here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2160650252
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
We have `HoodieStorage.createImmutableFileInPath` that can be utilized to ensure file creation atomicity if that is your purpose. BTW, if the file creation is atomic, then the `backupCfgPath` related logic can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994348457
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7df2acc0135dc8d64d8e542d435ee363afedb5f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2167066908
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
@danny0405 can you please review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2311832780
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Thanks @danny0405 for reviewing the PR https://github.com/apache/hudi/pull/13740
Since we have added all the guardrails to makesure that properties file is not corrupted, I'm thinking we still need this atomic rename to make sure that we didn't end up in empty property file (due to JVM crash or filesytem gets closed or due to other issues, even if we end up in empty file or corrupted file it gets handled by the prechecks added, but this rename will make sure that there will be no empty and no calls in the recoverIfNeeded method). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193594305
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a2477bdccc790d64c46171f4e698b7133ce6a5b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865)
* 83ee9f980434afd65f7bd626ff123e52855e0799 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868)
* 725fe0b9b12286a2953ab50818478fdd33c71efd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193597960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 83ee9f980434afd65f7bd626ff123e52855e0799 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868)
* 725fe0b9b12286a2953ab50818478fdd33c71efd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2287001537
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
@danny0405 do you see any issues with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-2994347677
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7df2acc0135dc8d64d8e542d435ee363afedb5f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2188816891
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
1. The first instant when the properties file became empty, the backup file serve the readers.
2. When the next update came for properties file, we are checking if properties file is not present then recover from the backup file.
3. SInce properties file is present but empty, we won't recover from backup file and we will delete the backup file which contains the actual properties.
4. This create both properties and backup become empty.
5. This temp file logic will make sure that the original will be present if it has been populated correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13473:
URL: https://github.com/apache/hudi/pull/13473#issuecomment-3193634479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6400",
"triggerID" : "c7df2acc0135dc8d64d8e542d435ee363afedb5f",
"triggerType" : "PUSH"
}, {
"hash" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6413",
"triggerID" : "ca048517ee0d0e62b57bfc47a2c5e2efca99439e",
"triggerType" : "PUSH"
}, {
"hash" : "eee2065f1c05687b82a611f63de189438553533c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7864",
"triggerID" : "eee2065f1c05687b82a611f63de189438553533c",
"triggerType" : "PUSH"
}, {
"hash" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7865",
"triggerID" : "1a2477bdccc790d64c46171f4e698b7133ce6a5b",
"triggerType" : "PUSH"
}, {
"hash" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7868",
"triggerID" : "83ee9f980434afd65f7bd626ff123e52855e0799",
"triggerType" : "PUSH"
}, {
"hash" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869",
"triggerID" : "725fe0b9b12286a2953ab50818478fdd33c71efd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 725fe0b9b12286a2953ab50818478fdd33c71efd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2279209679
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah @danny0405 there are multiple issues that can happen right now due to file creation and post copy. Let say in hdfs, our table path is running out of quota and it reached it max. For now it reached it's limit in diskspace quota and with the create and copy of actual file directly, will end up in creation of 0 bytes properties file (Since there is no issue of namespace quota) and while copying, it will fail due to quota issue. And that's how original file will be present with 0 bytes. With this rename approach, all the intermittent quota issues or namenode latency issues will be fallback to temp file and original file will be populated once the temp file is populated correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288200767
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
> It looks like what is really problematic is the #recoverIfNeeded, it just check the file existence of the cfgPath but not the checksum, which could incur empty files like you mentioned.
Yeah exactly. I can address this is next PR
> I did't see the issues, just try to understand why this is needed, we already have backup file and checksum validation for cfg file, when the file is empty, the checksum would be invalid, and the reader would just read from backup file.
Just think if this issue happened twice in a row in a different manner.
first actual pros will get empty
Next one since actual file is present and we will drop the backup file which contains actual props
and while populating the backup again we will end up in 0 bytes for the back up as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2312283806
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah we can rely on the pre-checks added.
Cool let the close the PR
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah we can rely on the pre-checks added.
Cool let me close the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2312283806
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
Yeah we can rely on the pre-checks added.
Cool let me close the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the creation of properties file atomic during modification [hudi]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #13473:
URL: https://github.com/apache/hudi/pull/13473#discussion_r2288954442
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -504,6 +505,7 @@ public static void recover(HoodieStorage storage, StoragePath metadataFolder) th
private static void modify(HoodieStorage storage, StoragePath metadataFolder, Properties modifyProps, BiConsumer<Properties, Properties> modifyFn) {
StoragePath cfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE);
StoragePath backupCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ StoragePath tempCfgPath = new StoragePath(metadataFolder, HOODIE_PROPERTIES_FILE + HOODIE_TEMP_FILE_SUFFIX);
Review Comment:
@danny0405 addressed the first issue in this PR https://github.com/apache/hudi/pull/13740
can you please review that as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org