[jira] [Updated] (TEZ-2882) Consider improving fetch failure handling

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Balamohan updated TEZ-2882:
----------------------------------
    Attachment: TEZ-2882.1.patch

Changes:
- Reduced abortFailureLimit from 30 to 15. This should be fine for detecting read/connect issues. But could be aggressive when server returns 500 internal server (e.g server had disk issues and was able to read index file properly. But when streaming real contents, it encountered disk issues and ends up throwing 500 internal server error. In such cases, reducing this value from 30 to 15 might cause little more aggressive failures.  This should be ok, as in case of 500 internal server, there is hardly a chance for the server to report healthy output).
- Added ability to detect failure rates since last progress. Task health is checked based on this and this would improve the accuracy of whether consumer has to be restarted or source has to be restarted. Also, consumer would be restarted only when errors have happened across 20% of the hosts (e.  Failing to fetch from 1 host, but succeeded from others – it’s like that 1 host's problem. Failing to fetch from a large number of hosts, it’s likely caused by the consumer).
- Added set of tests for this. Added a simple test for checking penalty as well.

Not covered in this:
- In case producer host gets restarted, consumer could get 404 error. This is handled in the same way as other type of read exceptions (e.g 500 internal server error, or shuffle header mismatch etc). Ideally, it might be good to restart the producer as soon as possible in AM side on observing 404 (instead of waiting for the retry cycle). This can be addressed in separate ticket, as it would not cause any job hang currently.

[~sseth] - Please review when you find time.

> Consider improving fetch failure handling
> -----------------------------------------
>
>                 Key: TEZ-2882
>                 URL: https://issues.apache.org/jira/browse/TEZ-2882
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: TEZ-2882.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)