You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Ahmed Hussein (JIRA)" <ji...@apache.org> on 2019/05/30 17:48:00 UTC
[jira] [Issue Comment Deleted] (TEZ-4067) Tez Speculation decision
is calculated on each update by the dispatcher
[ https://issues.apache.org/jira/browse/TEZ-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Hussein updated TEZ-4067:
-------------------------------
Comment: was deleted
(was: TEZ-1897)
> Tez Speculation decision is calculated on each update by the dispatcher
> -----------------------------------------------------------------------
>
> Key: TEZ-4067
> URL: https://issues.apache.org/jira/browse/TEZ-4067
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Ahmed Hussein
> Assignee: Ahmed Hussein
> Priority: Minor
> Attachments: TEZ-4067.001.patch, TEZ-4067.002.patch
>
>
> LegacySpeculator is an object field in VertexImpl. Therefore, all events are handled synchronously by the caller (dispatcher). This implies the following:
> # the dispatcher spends long time executing updateStatus as it needs to check the runtime estimation of the tezAttempts within the vertex.
> # the speculator is per stage: lunching a speculation may not the optimum decision. Ideally, based on resources, speculated tasks should be the ones with slowest progress.
> # the time between speculation is skewed because there is a big delay for the dispatcher to complete a full cycle. Also, speculation will be more aggressive compared to MR because MR waits for "soonest.retry.after.speculate" whenever a task is speculated. On the other hand, Tez speculates more tasks as it processes stages in parallel.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)