You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by abhinav gupta <ab...@yahoo.com> on 2013/05/24 09:43:32 UTC
Abort a job when a counter reaches to a threshold
Hi,
While running a map-reduce job, that has only mappers, I have a counter that counts the number of failed documents .And after all the
mappers are done, I want the job to fail if the total number of failed
documents are above a fixed fraction. ( I need it in the end because I
don't know the total number of documents initially). How can I achieve
this without implementing a reduce just for this ?
I know that there are task level cleanup method. But is there any job level cleanup method, that can be used to perform this after all the
tasks are done ?
Thanks
Abhinav
Re: Abort a job when a counter reaches to a threshold
Posted by Harsh J <ha...@cloudera.com>.
Yes there is a job level end-point upon success via OutputCommitter:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputCommitter.html#commitJob(org.apache.hadoop.mapreduce.JobContext)
On Fri, May 24, 2013 at 1:13 PM, abhinav gupta <ab...@yahoo.com> wrote:
> Hi,
>
> While running a map-reduce job, that has only mappers, I have a counter that
> counts the number of failed documents .And after all the mappers are done, I
> want the job to fail if the total number of failed documents are above a
> fixed fraction. ( I need it in the end because I don't know the total number
> of documents initially). How can I achieve this without implementing a
> reduce just for this ?
> I know that there are task level cleanup method. But is there any job level
> cleanup method, that can be used to perform this after all the tasks are
> done ?
>
> Thanks
> Abhinav
--
Harsh J
Re: Abort a job when a counter reaches to a threshold
Posted by Harsh J <ha...@cloudera.com>.
Yes there is a job level end-point upon success via OutputCommitter:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputCommitter.html#commitJob(org.apache.hadoop.mapreduce.JobContext)
On Fri, May 24, 2013 at 1:13 PM, abhinav gupta <ab...@yahoo.com> wrote:
> Hi,
>
> While running a map-reduce job, that has only mappers, I have a counter that
> counts the number of failed documents .And after all the mappers are done, I
> want the job to fail if the total number of failed documents are above a
> fixed fraction. ( I need it in the end because I don't know the total number
> of documents initially). How can I achieve this without implementing a
> reduce just for this ?
> I know that there are task level cleanup method. But is there any job level
> cleanup method, that can be used to perform this after all the tasks are
> done ?
>
> Thanks
> Abhinav
--
Harsh J
Re: Abort a job when a counter reaches to a threshold
Posted by Harsh J <ha...@cloudera.com>.
Yes there is a job level end-point upon success via OutputCommitter:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputCommitter.html#commitJob(org.apache.hadoop.mapreduce.JobContext)
On Fri, May 24, 2013 at 1:13 PM, abhinav gupta <ab...@yahoo.com> wrote:
> Hi,
>
> While running a map-reduce job, that has only mappers, I have a counter that
> counts the number of failed documents .And after all the mappers are done, I
> want the job to fail if the total number of failed documents are above a
> fixed fraction. ( I need it in the end because I don't know the total number
> of documents initially). How can I achieve this without implementing a
> reduce just for this ?
> I know that there are task level cleanup method. But is there any job level
> cleanup method, that can be used to perform this after all the tasks are
> done ?
>
> Thanks
> Abhinav
--
Harsh J
Re: Abort a job when a counter reaches to a threshold
Posted by Harsh J <ha...@cloudera.com>.
Yes there is a job level end-point upon success via OutputCommitter:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputCommitter.html#commitJob(org.apache.hadoop.mapreduce.JobContext)
On Fri, May 24, 2013 at 1:13 PM, abhinav gupta <ab...@yahoo.com> wrote:
> Hi,
>
> While running a map-reduce job, that has only mappers, I have a counter that
> counts the number of failed documents .And after all the mappers are done, I
> want the job to fail if the total number of failed documents are above a
> fixed fraction. ( I need it in the end because I don't know the total number
> of documents initially). How can I achieve this without implementing a
> reduce just for this ?
> I know that there are task level cleanup method. But is there any job level
> cleanup method, that can be used to perform this after all the tasks are
> done ?
>
> Thanks
> Abhinav
--
Harsh J