[jira] [Commented] (HADOOP-12844) Recover when S3A fails on IOException

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233493#comment-15233493 ] 

Steve Loughran commented on HADOOP-12844:
-----------------------------------------

with HADOOP-12444, the EOF exception on read is mapped to -1, so that part of the patch is now done.

The second aspect, {{read(array, offset, len)}} having EOF -> -1, other IOE -> re-open on a failure isn't in there, but as the length is checked earlier on, the codepath will only get explored if the file was overwritten during use, and the replacement was shorter.

We fixed the PositionedReadable handling of EOFs in HADOOP-12994; this JIRA is an evolution, including better resilience.

Can you update it against the latest code & I'll review.
# the IOE handling should include the string value of the IOE in the info-level message, the full exception as the argument. why? because if something is failing, repeatedly, you will want to know. Note that you can use SLF4J log formatting for ease of construction of the log message.

#  I'd really like see some tests here. Is there any way to do this. I'd imagine: create the file, open it, delete the file, then attempt the read. then do something similar when the existing object is overwritten with a shorter object. 

> Recover when S3A fails on IOException
> -------------------------------------
>
>                 Key: HADOOP-12844
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12844
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.1, 2.7.2
>            Reporter: Pieter Reuse
>            Assignee: Pieter Reuse
>         Attachments: HADOOP-12844.001.patch
>
>
> This simple patch catches IOExceptions in S3AInputStream.read(byte[] buf, int off, int len) and reopens the connection on the same location as it was before the exception.
> This is similar to the functionality introduced in S3N in [HADOOP-6254|https://issues.apache.org/jira/browse/HADOOP-6254], for exactly the same reason.
> Patch developed in cooperation with [~emres].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)