You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by 梁景明 <fu...@gmail.com> on 2009/10/21 05:28:16 UTC
How to skip fail map to done the job
hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
,and completed 128 map,
but only 2 map fail ,and its fail in my case is accepted ,but the job fail
,the last 128 map also fail,
i can't get any result.
i found that SkipBadRecords maybe work for my case,but i used it ,and it
doesnt work,maybe something
wrong,here is my code , how to skip the bad fail map ,thanks for any help
JobConf conf = new JobConf(ProductMR.class);
conf.setJobName("ProductMR");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Product.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setMapOutputCompressorClass(DefaultCodec.class);
conf.setInputFormat(SequenceFileInputFormat.class);
conf.setOutputFormat(SequenceFileOutputFormat.class);
String objpath = "abc1";
SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
String output = "abc";
SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
JobClient.runJob(conf);
Re: How to skip fail map to done the job
Posted by 梁景明 <fu...@gmail.com>.
thank you ,it works
2009/10/21 Amogh Vasekar <am...@yahoo-inc.com>
> For skipping failed tasks try : mapred.max.map.failures.percent
>
> Amogh
> On 10/21/09 8:58 AM, "梁景明" <fu...@gmail.com> wrote:
>
> hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
> ,and completed 128 map,
> but only 2 map fail ,and its fail in my case is accepted ,but the job fail
> ,the last 128 map also fail,
> i can't get any result.
> i found that SkipBadRecords maybe work for my case,but i used it ,and it
> doesnt work,maybe something
> wrong,here is my code , how to skip the bad fail map ,thanks for any help
>
> JobConf conf = new JobConf(ProductMR.class);
> conf.setJobName("ProductMR");
> conf.setOutputKeyClass(Text.class);
> conf.setOutputValueClass(Product.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setMapOutputCompressorClass(DefaultCodec.class);
> conf.setInputFormat(SequenceFileInputFormat.class);
> conf.setOutputFormat(SequenceFileOutputFormat.class);
> String objpath = "abc1";
> SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
> SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
> SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
> SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
> String output = "abc";
> SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
> JobClient.runJob(conf);
>
>
Re: How to skip fail map to done the job
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
For skipping failed tasks try : mapred.max.map.failures.percent
Amogh
On 10/21/09 8:58 AM, "梁景明" <fu...@gmail.com> wrote:
hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
,and completed 128 map,
but only 2 map fail ,and its fail in my case is accepted ,but the job fail
,the last 128 map also fail,
i can't get any result.
i found that SkipBadRecords maybe work for my case,but i used it ,and it
doesnt work,maybe something
wrong,here is my code , how to skip the bad fail map ,thanks for any help
JobConf conf = new JobConf(ProductMR.class);
conf.setJobName("ProductMR");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Product.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setMapOutputCompressorClass(DefaultCodec.class);
conf.setInputFormat(SequenceFileInputFormat.class);
conf.setOutputFormat(SequenceFileOutputFormat.class);
String objpath = "abc1";
SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
String output = "abc";
SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
JobClient.runJob(conf);