[jira] [Commented] (HADOOP-17458) S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException

["Prabhu Joseph (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597298#comment-17597298 ] 

Prabhu Joseph commented on HADOOP-17458:
----------------------------------------

[~stevel@apache.org] A flink job which reads data from S3 and intermittently few tasks fails with below exception. Does this patch will fix the below issue. Thanks.

{code}
Data read has a different length than the expected: dataLength=53427; expectedLength=65536; includeSkipped=true; in.getClass()=class com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
        at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)
        at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:93)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84)
        at com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:99)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:84)
        at org.apache.hadoop.fs.s3a.S3AInputStream.closeStream(S3AInputStream.java:529)
        at org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:490)
        at java.io.FilterInputStream.close(FilterInputStream.java:181)
        at org.apache.flink.fs.s3hadoop.common.HadoopDataInputStream.close(HadoopDataInputStream.java:91)
        at org.apache.flink.api.common.io.FileInputFormat.close(FileInputFormat.java:913)
        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:219)
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
        at java.lang.Thread.run(Thread.java:750)

{code}


> S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-17458
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17458
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.4.0
>            Reporter: Steve Loughran
>            Assignee: Bogdan Stolojan
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 3.3.2
>
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> A test run with network problems caught exceptions "com.amazonaws.SdkClientException: Data read has a different length than the expected:", which then escalated to failure.
> these should be recoverable if they are recognised as such. 
> translateException could do this. Yes, it would have to look @ the text, but as {{signifiesConnectionBroken()}} already does that for "Failed to sanitize XML document destined for handler class", it'd just be adding a new text string to look for.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org