[jira] [Comment Edited] (TEZ-1062) Create SimpleProcessor for processors that only need to implement the run method

["Mohammad Kamrul Islam (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977734#comment-13977734 ] 

Mohammad Kamrul Islam edited comment on TEZ-1062 at 4/23/14 2:44 AM:
---------------------------------------------------------------------

Thanks again [~bikassaha] for the review.
Need some clarifications:

bq. Secondly, we should call getContext().canCommit() only once ...

Struggling to find the correct ordering of two checks. How about this?

{noformat}
+   while (!getContext().canCommit()) {
+        Thread.sleep(100);
+    }
+   for (LogicalOutput output : outputs.values()) {
+        if ((output instanceof MROutput) && (((MROutput) output).isCommitRequired())) {
+          ((MROutput) output).commit();
+        }
+    }
{noformat}

Related question:
bq.  If any output fails to commit then we should abort all the outputs that needed commit.

I think already committed outputs can't be aborted. 

How to determine some output fails? when output.commit() throw exception?

By abort, do you mean to break the loop. 

EDITED:
bq. I think it makes sense to move this code into the postOp of SimpleProcessor.

MROutput is not accessible from Tez-runitme-library. If I include the tez-mapreduce (where MROutput resides) in the pom.xml of tez-runtime-library module, it creates a circular dependency . any thoughts?




was (Author: kamrul):
Thanks again [~bikassaha] for the review.
Need some clarifications:

bq. Secondly, we should call getContext().canCommit() only once ...

Struggling to find the correct ordering of two checks. How about this?

{noformat}
+   while (!getContext().canCommit()) {
+        Thread.sleep(100);
+    }
+   for (LogicalOutput output : outputs.values()) {
+        if ((output instanceof MROutput) && (((MROutput) output).isCommitRequired())) {
+          ((MROutput) output).commit();
+        }
+    }
{noformat}

Related question:
bq.  If any output fails to commit then we should abort all the outputs that needed commit.

I think already committed outputs can't be aborted. 

How to determine some output fails? when output.commit() throw exception?

By abort, do you mean to break the loop. 




> Create SimpleProcessor for processors that only need to implement the run method
> --------------------------------------------------------------------------------
>
>                 Key: TEZ-1062
>                 URL: https://issues.apache.org/jira/browse/TEZ-1062
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Mohammad Kamrul Islam
>         Attachments: TEZ-1062.1.patch
>
>
> The SimpleProcessor could take care of all things like starting input, committing outputs. It would handle no events, since simple processors dont need to handle inputs. Thus the user would only need to implement their custom task logic in a new execute() method.



--
This message was sent by Atlassian JIRA
(v6.2#6252)