You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2023/07/10 16:14:37 UTC
[GitHub] [hudi] lokeshj1703 opened a new pull request, #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
lokeshj1703 opened a new pull request, #9165:
URL: https://github.com/apache/hudi/pull/9165
### Change Logs
Currently we do not throw an exception if invalid data file deletion fails.
```
fileSystem.delete(new Path(partitionFilePair.getValue()), false)
```
The result of the delete call is not checked today. The PR adds a check for it and throws exception in case delete fails.
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9165:
URL: https://github.com/apache/hudi/pull/9165#issuecomment-1629377911
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18456",
"triggerID" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0055f7d26380a87af8c29ec2abb0d4d1e0606d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18456)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9165:
URL: https://github.com/apache/hudi/pull/9165#discussion_r1260403670
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -688,7 +688,10 @@ private void deleteInvalidFilesByPartitions(HoodieEngineContext context, Map<Str
LOG.info("Deleting invalid data file=" + partitionFilePair);
// Delete
try {
- fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ boolean result = fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ if (!result) {
Review Comment:
oh, on re-attempt, do we know if we might try to delete a non-existant file ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9165:
URL: https://github.com/apache/hudi/pull/9165#issuecomment-1629801470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18456",
"triggerID" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0055f7d26380a87af8c29ec2abb0d4d1e0606d2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18456)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #9165:
URL: https://github.com/apache/hudi/pull/9165#issuecomment-1629273314
cc @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9165:
URL: https://github.com/apache/hudi/pull/9165#discussion_r1279066687
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -688,7 +688,10 @@ private void deleteInvalidFilesByPartitions(HoodieEngineContext context, Map<Str
LOG.info("Deleting invalid data file=" + partitionFilePair);
// Delete
try {
- fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ boolean result = fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ if (!result) {
Review Comment:
This may not work as again it could fail due to same set of invalid files (no change in markers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
URL: https://github.com/apache/hudi/pull/9165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9165:
URL: https://github.com/apache/hudi/pull/9165#discussion_r1279053479
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -688,7 +688,10 @@ private void deleteInvalidFilesByPartitions(HoodieEngineContext context, Map<Str
LOG.info("Deleting invalid data file=" + partitionFilePair);
// Delete
try {
- fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ boolean result = fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ if (!result) {
Review Comment:
Doing fs.exists check everytime may be ineffcient. Why not ignore i.e catch and just log a warning? If the intention is to delete and file does not already exist then what's point of retrying?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9165:
URL: https://github.com/apache/hudi/pull/9165#issuecomment-1629363398
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0055f7d26380a87af8c29ec2abb0d4d1e0606d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0055f7d26380a87af8c29ec2abb0d4d1e0606d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9165: [HUDI-6517] Throw error if deletion of invalid data file fails
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9165:
URL: https://github.com/apache/hudi/pull/9165#discussion_r1260496231
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -688,7 +688,10 @@ private void deleteInvalidFilesByPartitions(HoodieEngineContext context, Map<Str
LOG.info("Deleting invalid data file=" + partitionFilePair);
// Delete
try {
- fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ boolean result = fileSystem.delete(new Path(partitionFilePair.getValue()), false);
+ if (!result) {
Review Comment:
maybe it is more safe to check the existence before deletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org