[jira] [Resolved] (TEZ-4381) Speed up TestSecureShuffle

["László Bodor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

László Bodor resolved TEZ-4381.
-------------------------------
    Resolution: Fixed

> Speed up TestSecureShuffle
> --------------------------
>
>                 Key: TEZ-4381
>                 URL: https://issues.apache.org/jira/browse/TEZ-4381
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: László Bodor
>            Assignee: Mark Bathori
>            Priority: Major
>             Fix For: 0.9.3, 0.10.2
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> TestSecureShuffle is very slow. We need to experiment with it.
> There is already an optimization like:
> {code}
>     conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT, 2);
> {code}
> Maybe it is worth checking if lowering the number of failed attempts can help. In case of shuffle failures, sooner or later the reducer and mapper task is retried, so lowering:
> {code}
> TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
> {code}
> might help, theoretically, as we want the DAG to fail as fast as possible.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)