You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Håvard Wahl Kongsgård <ha...@gmail.com> on 2013/01/05 10:53:55 UTC
Skipping entire task
Hi, hadoop can skip bad records
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-code.
But it is also possible to skip entire tasks?
-Håvard
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or something for that purpose. Any other ideas?
-Håvard
On Sat, Jan 5, 2013 at 3:14 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Thanks, I was unaware of mapred.max.map.failures.percent
-Håvard
On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <ha...@cloudera.com> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <ha...@gmail.com> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>> Job myJob = new MyJob(getConf());
>>>> myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>> //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
>From a user perspective, at a high level, the mapreduce package can be
thought of as having user facing client code that can be invoked, extended
etc as applicable from client programs.
The mapred package is to be treated as internal to the mapreduce system,
and shouldn't directly be used unless no alternative in the mapreduce
package is available.
Thanks
Hemanth
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Oleg,
Mapreduce 0.20.* api has the support for both 0.19 api (which is
MapRed package in which your mapper should extend MRBase and implements
Mapper) and it has the new api as well (which is MapReduce packages where
you directly extend Mapper).
As there are significant changes in 20 api may be for backward
compatibility mapred package is still in existence.
There are few classes which exist in 19 api and those are not
supported in 0.20.* version.
Best,
Mahesh Balija,
Calsoft Labs.
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Sandeep Dukkipati <sa...@gmail.com>.
I am a little new to the hadoop world. But based on my readings and
understanding thus far, there is not much functionality difference. Only
difference is new API allows you to implement push and pull mechanisms in
your map/reduce tasks as against push in old API. Mapper has been changed
to super class as against interface, so more functionality could be added
without affecting clients.
Please see,
http://stackoverflow.com/questions/7598422/is-it-better-to-use-the-mapred-or-the-mapreduce-package-to-create-a-hadoop-job
On Mon, Jan 7, 2013 at 10:14 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Sandeep Dukkipati <sa...@gmail.com>.
I am a little new to the hadoop world. But based on my readings and
understanding thus far, there is not much functionality difference. Only
difference is new API allows you to implement push and pull mechanisms in
your map/reduce tasks as against push in old API. Mapper has been changed
to super class as against interface, so more functionality could be added
without affecting clients.
Please see,
http://stackoverflow.com/questions/7598422/is-it-better-to-use-the-mapred-or-the-mapreduce-package-to-create-a-hadoop-job
On Mon, Jan 7, 2013 at 10:14 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
>From a user perspective, at a high level, the mapreduce package can be
thought of as having user facing client code that can be invoked, extended
etc as applicable from client programs.
The mapred package is to be treated as internal to the mapreduce system,
and shouldn't directly be used unless no alternative in the mapreduce
package is available.
Thanks
Hemanth
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Oleg,
Mapreduce 0.20.* api has the support for both 0.19 api (which is
MapRed package in which your mapper should extend MRBase and implements
Mapper) and it has the new api as well (which is MapReduce packages where
you directly extend Mapper).
As there are significant changes in 20 api may be for backward
compatibility mapred package is still in existence.
There are few classes which exist in 19 api and those are not
supported in 0.20.* version.
Best,
Mahesh Balija,
Calsoft Labs.
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Sandeep Dukkipati <sa...@gmail.com>.
I am a little new to the hadoop world. But based on my readings and
understanding thus far, there is not much functionality difference. Only
difference is new API allows you to implement push and pull mechanisms in
your map/reduce tasks as against push in old API. Mapper has been changed
to super class as against interface, so more functionality could be added
without affecting clients.
Please see,
http://stackoverflow.com/questions/7598422/is-it-better-to-use-the-mapred-or-the-mapreduce-package-to-create-a-hadoop-job
On Mon, Jan 7, 2013 at 10:14 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
>From a user perspective, at a high level, the mapreduce package can be
thought of as having user facing client code that can be invoked, extended
etc as applicable from client programs.
The mapred package is to be treated as internal to the mapreduce system,
and shouldn't directly be used unless no alternative in the mapreduce
package is available.
Thanks
Hemanth
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Oleg,
Mapreduce 0.20.* api has the support for both 0.19 api (which is
MapRed package in which your mapper should extend MRBase and implements
Mapper) and it has the new api as well (which is MapReduce packages where
you directly extend Mapper).
As there are significant changes in 20 api may be for backward
compatibility mapred package is still in existence.
There are few classes which exist in 19 api and those are not
supported in 0.20.* version.
Best,
Mahesh Balija,
Calsoft Labs.
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Sandeep Dukkipati <sa...@gmail.com>.
I am a little new to the hadoop world. But based on my readings and
understanding thus far, there is not much functionality difference. Only
difference is new API allows you to implement push and pull mechanisms in
your map/reduce tasks as against push in old API. Mapper has been changed
to super class as against interface, so more functionality could be added
without affecting clients.
Please see,
http://stackoverflow.com/questions/7598422/is-it-better-to-use-the-mapred-or-the-mapreduce-package-to-create-a-hadoop-job
On Mon, Jan 7, 2013 at 10:14 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Oleg,
Mapreduce 0.20.* api has the support for both 0.19 api (which is
MapRed package in which your mapper should extend MRBase and implements
Mapper) and it has the new api as well (which is MapReduce packages where
you directly extend Mapper).
As there are significant changes in 20 api may be for backward
compatibility mapred package is still in existence.
There are few classes which exist in 19 api and those are not
supported in 0.20.* version.
Best,
Mahesh Balija,
Calsoft Labs.
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Re: Differences between 'mapped' and 'mapreduce' packages
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
>From a user perspective, at a high level, the mapreduce package can be
thought of as having user facing client code that can be invoked, extended
etc as applicable from client programs.
The mapred package is to be treated as internal to the mapreduce system,
and shouldn't directly be used unless no alternative in the mapreduce
package is available.
Thanks
Hemanth
On Mon, Jan 7, 2013 at 11:44 PM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> What is the differences between the two?
> It seems like MR job could be configured using one of the other (e.g,
> "extends MapReduceBase implements Mapper" of "extends Mapper")
>
> Cheers
> Oleg
Differences between 'mapped' and 'mapreduce' packages
Posted by Oleg Zhurakousky <ol...@gmail.com>.
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, "extends MapReduceBase implements Mapper" of "extends Mapper")
Cheers
Oleg
Differences between 'mapped' and 'mapreduce' packages
Posted by Oleg Zhurakousky <ol...@gmail.com>.
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, "extends MapReduceBase implements Mapper" of "extends Mapper")
Cheers
Oleg
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Thanks, I was unaware of mapred.max.map.failures.percent
-Håvard
On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <ha...@cloudera.com> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <ha...@gmail.com> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>> Job myJob = new MyJob(getConf());
>>>> myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>> //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Differences between 'mapped' and 'mapreduce' packages
Posted by Oleg Zhurakousky <ol...@gmail.com>.
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, "extends MapReduceBase implements Mapper" of "extends Mapper")
Cheers
Oleg
Differences between 'mapped' and 'mapreduce' packages
Posted by Oleg Zhurakousky <ol...@gmail.com>.
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, "extends MapReduceBase implements Mapper" of "extends Mapper")
Cheers
Oleg
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Thanks, I was unaware of mapred.max.map.failures.percent
-Håvard
On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <ha...@cloudera.com> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <ha...@gmail.com> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>> Job myJob = new MyJob(getConf());
>>>> myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>> //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Thanks, I was unaware of mapred.max.map.failures.percent
-Håvard
On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <ha...@cloudera.com> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <ha...@gmail.com> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <yh...@thoughtworks.com> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>> Job myJob = new MyJob(getConf());
>>>> myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>> //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Harsh J <ha...@cloudera.com>.
You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).
On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
<ha...@gmail.com> wrote:
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>
> yes, and I simply want to skip them if they fail more than x
> times(after all this is big data :) ).
>
> -Håvard
>
> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
>> Hi,
>>
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>>
>> Thanks
>> hemanth
>>
>>
>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>
>>> Thinking here... if you submitted the task programmatically you should be
>>> able to capture the failure of the task and gracefully move past it to
>>> your
>>> next tasks.
>>>
>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>> java jar, and your main class implements Tool. That code has the
>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>
>>> try{
>>> Job myJob = new MyJob(getConf());
>>> myJob.submitAndWait();
>>> }catch(Exception uhhohh){
>>> //Deal with the issue and move on
>>> }
>>> Job myNextJob = new MyNextJob(getConf());
>>> myNextJob.submit();
>>>
>>> Just pseudo code there to demonstrate my thought.
>>>
>>> David
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>> Sent: Saturday, January 05, 2013 4:54 PM
>>> To: user
>>> Subject: Skipping entire task
>>>
>>> Hi, hadoop can skip bad records
>>>
>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>> ode.
>>> But it is also possible to skip entire tasks?
>>>
>>> -Håvard
>>>
>>> --
>>> Håvard Wahl Kongsgård
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>>
>>> http://havard.security-review.net/
>>>
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
--
Harsh J
Re: Skipping entire task
Posted by Harsh J <ha...@cloudera.com>.
You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).
On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
<ha...@gmail.com> wrote:
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>
> yes, and I simply want to skip them if they fail more than x
> times(after all this is big data :) ).
>
> -Håvard
>
> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
>> Hi,
>>
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>>
>> Thanks
>> hemanth
>>
>>
>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>
>>> Thinking here... if you submitted the task programmatically you should be
>>> able to capture the failure of the task and gracefully move past it to
>>> your
>>> next tasks.
>>>
>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>> java jar, and your main class implements Tool. That code has the
>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>
>>> try{
>>> Job myJob = new MyJob(getConf());
>>> myJob.submitAndWait();
>>> }catch(Exception uhhohh){
>>> //Deal with the issue and move on
>>> }
>>> Job myNextJob = new MyNextJob(getConf());
>>> myNextJob.submit();
>>>
>>> Just pseudo code there to demonstrate my thought.
>>>
>>> David
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>> Sent: Saturday, January 05, 2013 4:54 PM
>>> To: user
>>> Subject: Skipping entire task
>>>
>>> Hi, hadoop can skip bad records
>>>
>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>> ode.
>>> But it is also possible to skip entire tasks?
>>>
>>> -Håvard
>>>
>>> --
>>> Håvard Wahl Kongsgård
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>>
>>> http://havard.security-review.net/
>>>
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
--
Harsh J
Re: Skipping entire task
Posted by Harsh J <ha...@cloudera.com>.
You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).
On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
<ha...@gmail.com> wrote:
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>
> yes, and I simply want to skip them if they fail more than x
> times(after all this is big data :) ).
>
> -Håvard
>
> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
>> Hi,
>>
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>>
>> Thanks
>> hemanth
>>
>>
>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>
>>> Thinking here... if you submitted the task programmatically you should be
>>> able to capture the failure of the task and gracefully move past it to
>>> your
>>> next tasks.
>>>
>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>> java jar, and your main class implements Tool. That code has the
>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>
>>> try{
>>> Job myJob = new MyJob(getConf());
>>> myJob.submitAndWait();
>>> }catch(Exception uhhohh){
>>> //Deal with the issue and move on
>>> }
>>> Job myNextJob = new MyNextJob(getConf());
>>> myNextJob.submit();
>>>
>>> Just pseudo code there to demonstrate my thought.
>>>
>>> David
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>> Sent: Saturday, January 05, 2013 4:54 PM
>>> To: user
>>> Subject: Skipping entire task
>>>
>>> Hi, hadoop can skip bad records
>>>
>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>> ode.
>>> But it is also possible to skip entire tasks?
>>>
>>> -Håvard
>>>
>>> --
>>> Håvard Wahl Kongsgård
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>>
>>> http://havard.security-review.net/
>>>
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
--
Harsh J
Re: Skipping entire task
Posted by Harsh J <ha...@cloudera.com>.
You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).
On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
<ha...@gmail.com> wrote:
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>
> yes, and I simply want to skip them if they fail more than x
> times(after all this is big data :) ).
>
> -Håvard
>
> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
> <yh...@thoughtworks.com> wrote:
>> Hi,
>>
>> Are tasks being executed multiple times due to failures? Sorry, it was not
>> very clear from your question.
>>
>> Thanks
>> hemanth
>>
>>
>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>>
>>> Thinking here... if you submitted the task programmatically you should be
>>> able to capture the failure of the task and gracefully move past it to
>>> your
>>> next tasks.
>>>
>>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>>> java jar, and your main class implements Tool. That code has the
>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>
>>> try{
>>> Job myJob = new MyJob(getConf());
>>> myJob.submitAndWait();
>>> }catch(Exception uhhohh){
>>> //Deal with the issue and move on
>>> }
>>> Job myNextJob = new MyNextJob(getConf());
>>> myNextJob.submit();
>>>
>>> Just pseudo code there to demonstrate my thought.
>>>
>>> David
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>>> Sent: Saturday, January 05, 2013 4:54 PM
>>> To: user
>>> Subject: Skipping entire task
>>>
>>> Hi, hadoop can skip bad records
>>>
>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>> ode.
>>> But it is also possible to skip entire tasks?
>>>
>>> -Håvard
>>>
>>> --
>>> Håvard Wahl Kongsgård
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>>
>>> http://havard.security-review.net/
>>>
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
--
Harsh J
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
yes, and I simply want to skip them if they fail more than x
times(after all this is big data :) ).
-Håvard
On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
>
> Thanks
> hemanth
>
>
> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>
>> Thinking here... if you submitted the task programmatically you should be
>> able to capture the failure of the task and gracefully move past it to
>> your
>> next tasks.
>>
>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>> java jar, and your main class implements Tool. That code has the
>> responsibility to submit a series of jobs to hadoop, something like this:
>>
>> try{
>> Job myJob = new MyJob(getConf());
>> myJob.submitAndWait();
>> }catch(Exception uhhohh){
>> //Deal with the issue and move on
>> }
>> Job myNextJob = new MyNextJob(getConf());
>> myNextJob.submit();
>>
>> Just pseudo code there to demonstrate my thought.
>>
>> David
>>
>>
>>
>> -----Original Message-----
>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>> Sent: Saturday, January 05, 2013 4:54 PM
>> To: user
>> Subject: Skipping entire task
>>
>> Hi, hadoop can skip bad records
>>
>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>> ode.
>> But it is also possible to skip entire tasks?
>>
>> -Håvard
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>>
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
yes, and I simply want to skip them if they fail more than x
times(after all this is big data :) ).
-Håvard
On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
>
> Thanks
> hemanth
>
>
> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>
>> Thinking here... if you submitted the task programmatically you should be
>> able to capture the failure of the task and gracefully move past it to
>> your
>> next tasks.
>>
>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>> java jar, and your main class implements Tool. That code has the
>> responsibility to submit a series of jobs to hadoop, something like this:
>>
>> try{
>> Job myJob = new MyJob(getConf());
>> myJob.submitAndWait();
>> }catch(Exception uhhohh){
>> //Deal with the issue and move on
>> }
>> Job myNextJob = new MyNextJob(getConf());
>> myNextJob.submit();
>>
>> Just pseudo code there to demonstrate my thought.
>>
>> David
>>
>>
>>
>> -----Original Message-----
>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>> Sent: Saturday, January 05, 2013 4:54 PM
>> To: user
>> Subject: Skipping entire task
>>
>> Hi, hadoop can skip bad records
>>
>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>> ode.
>> But it is also possible to skip entire tasks?
>>
>> -Håvard
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>>
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
yes, and I simply want to skip them if they fail more than x
times(after all this is big data :) ).
-Håvard
On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
>
> Thanks
> hemanth
>
>
> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>
>> Thinking here... if you submitted the task programmatically you should be
>> able to capture the failure of the task and gracefully move past it to
>> your
>> next tasks.
>>
>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>> java jar, and your main class implements Tool. That code has the
>> responsibility to submit a series of jobs to hadoop, something like this:
>>
>> try{
>> Job myJob = new MyJob(getConf());
>> myJob.submitAndWait();
>> }catch(Exception uhhohh){
>> //Deal with the issue and move on
>> }
>> Job myNextJob = new MyNextJob(getConf());
>> myNextJob.submit();
>>
>> Just pseudo code there to demonstrate my thought.
>>
>> David
>>
>>
>>
>> -----Original Message-----
>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>> Sent: Saturday, January 05, 2013 4:54 PM
>> To: user
>> Subject: Skipping entire task
>>
>> Hi, hadoop can skip bad records
>>
>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>> ode.
>> But it is also possible to skip entire tasks?
>>
>> -Håvard
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>>
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
yes, and I simply want to skip them if they fail more than x
times(after all this is big data :) ).
-Håvard
On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
<yh...@thoughtworks.com> wrote:
> Hi,
>
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
>
> Thanks
> hemanth
>
>
> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
>>
>> Thinking here... if you submitted the task programmatically you should be
>> able to capture the failure of the task and gracefully move past it to
>> your
>> next tasks.
>>
>> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
>> java jar, and your main class implements Tool. That code has the
>> responsibility to submit a series of jobs to hadoop, something like this:
>>
>> try{
>> Job myJob = new MyJob(getConf());
>> myJob.submitAndWait();
>> }catch(Exception uhhohh){
>> //Deal with the issue and move on
>> }
>> Job myNextJob = new MyNextJob(getConf());
>> myNextJob.submit();
>>
>> Just pseudo code there to demonstrate my thought.
>>
>> David
>>
>>
>>
>> -----Original Message-----
>> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
>> Sent: Saturday, January 05, 2013 4:54 PM
>> To: user
>> Subject: Skipping entire task
>>
>> Hi, hadoop can skip bad records
>>
>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>> ode.
>> But it is also possible to skip entire tasks?
>>
>> -Håvard
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>>
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
Are tasks being executed multiple times due to failures? Sorry, it was not
very clear from your question.
Thanks
hemanth
On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
>
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
>
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or something for that purpose. Any other ideas?
-Håvard
On Sat, Jan 5, 2013 at 3:14 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or something for that purpose. Any other ideas?
-Håvard
On Sat, Jan 5, 2013 at 3:14 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
Re: Skipping entire task
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
Are tasks being executed multiple times due to failures? Sorry, it was not
very clear from your question.
Thanks
hemanth
On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
>
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
>
Re: Skipping entire task
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
Are tasks being executed multiple times due to failures? Sorry, it was not
very clear from your question.
Thanks
hemanth
On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
>
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
>
Re: Skipping entire task
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
Are tasks being executed multiple times due to failures? Sorry, it was not
very clear from your question.
Thanks
hemanth
On Sat, Jan 5, 2013 at 7:44 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
>
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
>
Re: Skipping entire task
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or something for that purpose. Any other ideas?
-Håvard
On Sat, Jan 5, 2013 at 3:14 PM, David Parks <da...@yahoo.com> wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task and gracefully move past it to your
> next tasks.
>
> To say it in a long-winded way: Let's say you submit a job to Hadoop, a
> java jar, and your main class implements Tool. That code has the
> responsibility to submit a series of jobs to hadoop, something like this:
>
> try{
> Job myJob = new MyJob(getConf());
> myJob.submitAndWait();
> }catch(Exception uhhohh){
> //Deal with the issue and move on
> }
> Job myNextJob = new MyNextJob(getConf());
> myNextJob.submit();
>
> Just pseudo code there to demonstrate my thought.
>
> David
>
>
>
> -----Original Message-----
> From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
> Sent: Saturday, January 05, 2013 4:54 PM
> To: user
> Subject: Skipping entire task
>
> Hi, hadoop can skip bad records
> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
> ode.
> But it is also possible to skip entire tasks?
>
> -Håvard
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/
>
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
RE: Skipping entire task
Posted by David Parks <da...@yahoo.com>.
Thinking here... if you submitted the task programmatically you should be
able to capture the failure of the task and gracefully move past it to your
next tasks.
To say it in a long-winded way: Let's say you submit a job to Hadoop, a
java jar, and your main class implements Tool. That code has the
responsibility to submit a series of jobs to hadoop, something like this:
try{
Job myJob = new MyJob(getConf());
myJob.submitAndWait();
}catch(Exception uhhohh){
//Deal with the issue and move on
}
Job myNextJob = new MyNextJob(getConf());
myNextJob.submit();
Just pseudo code there to demonstrate my thought.
David
-----Original Message-----
From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
Sent: Saturday, January 05, 2013 4:54 PM
To: user
Subject: Skipping entire task
Hi, hadoop can skip bad records
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
ode.
But it is also possible to skip entire tasks?
-Håvard
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
RE: Skipping entire task
Posted by David Parks <da...@yahoo.com>.
Thinking here... if you submitted the task programmatically you should be
able to capture the failure of the task and gracefully move past it to your
next tasks.
To say it in a long-winded way: Let's say you submit a job to Hadoop, a
java jar, and your main class implements Tool. That code has the
responsibility to submit a series of jobs to hadoop, something like this:
try{
Job myJob = new MyJob(getConf());
myJob.submitAndWait();
}catch(Exception uhhohh){
//Deal with the issue and move on
}
Job myNextJob = new MyNextJob(getConf());
myNextJob.submit();
Just pseudo code there to demonstrate my thought.
David
-----Original Message-----
From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
Sent: Saturday, January 05, 2013 4:54 PM
To: user
Subject: Skipping entire task
Hi, hadoop can skip bad records
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
ode.
But it is also possible to skip entire tasks?
-Håvard
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
RE: Skipping entire task
Posted by David Parks <da...@yahoo.com>.
Thinking here... if you submitted the task programmatically you should be
able to capture the failure of the task and gracefully move past it to your
next tasks.
To say it in a long-winded way: Let's say you submit a job to Hadoop, a
java jar, and your main class implements Tool. That code has the
responsibility to submit a series of jobs to hadoop, something like this:
try{
Job myJob = new MyJob(getConf());
myJob.submitAndWait();
}catch(Exception uhhohh){
//Deal with the issue and move on
}
Job myNextJob = new MyNextJob(getConf());
myNextJob.submit();
Just pseudo code there to demonstrate my thought.
David
-----Original Message-----
From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
Sent: Saturday, January 05, 2013 4:54 PM
To: user
Subject: Skipping entire task
Hi, hadoop can skip bad records
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
ode.
But it is also possible to skip entire tasks?
-Håvard
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/
RE: Skipping entire task
Posted by David Parks <da...@yahoo.com>.
Thinking here... if you submitted the task programmatically you should be
able to capture the failure of the task and gracefully move past it to your
next tasks.
To say it in a long-winded way: Let's say you submit a job to Hadoop, a
java jar, and your main class implements Tool. That code has the
responsibility to submit a series of jobs to hadoop, something like this:
try{
Job myJob = new MyJob(getConf());
myJob.submitAndWait();
}catch(Exception uhhohh){
//Deal with the issue and move on
}
Job myNextJob = new MyNextJob(getConf());
myNextJob.submit();
Just pseudo code there to demonstrate my thought.
David
-----Original Message-----
From: Håvard Wahl Kongsgård [mailto:haavard.kongsgaard@gmail.com]
Sent: Saturday, January 05, 2013 4:54 PM
To: user
Subject: Skipping entire task
Hi, hadoop can skip bad records
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
ode.
But it is also possible to skip entire tasks?
-Håvard
--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.security-review.net/