[jira] [Comment Edited] (TEZ-853) Support counters recovery

["Jeff Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113433#comment-14113433 ] 

Jeff Zhang edited comment on TEZ-853 at 8/28/14 7:06 AM:
---------------------------------------------------------

[~hitesh]
bq. Do TaskImpl and VertexImpl write counters to recovery but they are not used when restoring state? Should the counters be written or recovered from task attempts? If the latter, then we should not write them.

There's no counter written from VertexImpl and TaskImpl ( their counters are from TaskAttemptImpl ) Should we remove tezCounters from VertexFinishedProto and TaskFinishedProto ? since actually we don't use it in recovery 

bq. DAGImpl::restoreFromEvent does not seem to restore counters
DAG do not write any counters, its counters are all from TaskAttemptImpl. So that means as long as counters of TaskAttemptImpl is recovered, Counters of DAG is recovered.

bq. in a scenario where the dag finished is logged and all other events are dropped, I assume counters will be needed?
Yes, you are right.This is a special case. In this case we should write counters in DAGFinishedEvent and recover from it. ( will add it )




was (Author: zjffdu):
[~hitesh]
bq. Do TaskImpl and VertexImpl write counters to recovery but they are not used when restoring state? Should the counters be written or recovered from task attempts? If the latter, then we should not write them.

There's no counter written from VertexImpl and TaskImpl ( their counters are from TaskAttemptImpl )

bq. DAGImpl::restoreFromEvent does not seem to restore counters
DAG do not write any counters, its counters are all from TaskAttemptImpl. So that means as long as counters of TaskAttemptImpl is recovered, Counters of DAG is recovered.

bq. in a scenario where the dag finished is logged and all other events are dropped, I assume counters will be needed?
Yes, you are right.This is a special case. In this case we should write counters in DAGFinishedEvent and recover from it. ( will add it )



> Support counters recovery
> -------------------------
>
>                 Key: TEZ-853
>                 URL: https://issues.apache.org/jira/browse/TEZ-853
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Hitesh Shah
>            Assignee: Jeff Zhang
>         Attachments: Tez-853.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)