You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/05/19 06:29:00 UTC
[jira] [Work logged] (HADOOP-18242) ABFS Rename Failure when tracking metadata is in incomplete state
[ https://issues.apache.org/jira/browse/HADOOP-18242?focusedWorklogId=772292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-772292 ]
ASF GitHub Bot logged work on HADOOP-18242:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 19/May/22 06:28
Start Date: 19/May/22 06:28
Worklog Time Spent: 10m
Work Description: mehakmeet opened a new pull request, #4331:
URL: https://github.com/apache/hadoop/pull/4331
### Description of PR
ABFS rename fails intermittently when the Storage-blob tracking metadata is in an incomplete state, to mitigate this issue, we can do a HEAD call on the source file and then retry the rename op.
### How was this patch tested?
Region : us-west-2
`mvn -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale clean verify`
```
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2
```
```
[INFO] Results:
[INFO]
[WARNING] Tests run: 567, Failures: 0, Errors: 0, Skipped: 75
```
[ERROR] Failures:
[ERROR] ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:69->testReadWriteAndSeek:101 [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 0, Skipped: 41
```
The error seems unrelated to the patch, more of "server-side" issue in a paramter(biggest size) of a parameterised test.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
Issue Time Tracking
-------------------
Worklog Id: (was: 772292)
Remaining Estimate: 0h
Time Spent: 10m
> ABFS Rename Failure when tracking metadata is in incomplete state
> -----------------------------------------------------------------
>
> Key: HADOOP-18242
> URL: https://issues.apache.org/jira/browse/HADOOP-18242
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a node in the datacenter crashes while processing an operation, occasionally it can leave the Storage-internal blob tracking metadata in an incomplete state. We expect this to happen occasionally, and so all API’s are designed in such a way that if this incomplete state is observed on a blob, the situation is resolved before the current operation proceeds. However, this incident has exposed a bug specifically with the Rename API, where the incomplete state fails to resolve, leading to this incorrect failure. As a temporary mitigation, if any other operation is performed on this blob – GetBlobProperties, GetBlob, GetFileProperties, SetFileProperties, etc – it should resolve the incomplete state, and rename will no longer hit this issue.
> StackTrace:
> {code:java}
> 2022-03-22 17:52:19,789 DEBUG [regionserver/euwukwlss-hg50:16020.logRoller] services.AbfsClient: HttpRequest: 404,RenameDestinationParentPathNotFound,cid=ef5cbf0f-5d4a-4630-8a59-3d559077fc24,rid=35fef164-101f-000b-1b15-3ed818000000,sent=0,recv=212,PUT,https://euwqdaotdfdls03.dfs.core.windows.net/eykbssc/apps/hbase/data/oldWALs/euwukwlss-hg50.tdf.qa%252C16020%252C1647949929877.1647967939315?timeout=90 {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org