You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Ruiz, Pierre" <Pi...@analog.com> on 2017/11/10 12:49:53 UTC
mpareduce.job.reduces greater than 1 crashes TotalOrderPartitioner
and/or RandomInputSampler
First time poster, I hope I'm not breaking any rule and if I am please tell me :)
I am trying to perform "total order" sorting on data previously stored on a pseudo-distributed HDFS (running with YARN) into a SequenceFile (keys: IntWritable, vals: Text). The SequenceFile, stored at /cnts/part-r-00000 to part-r-00006, contains 44 records altogether.
bin/hdfs dfs -text /cnts* prints 44 lines like:
2 314 | 12 | 21
2 700 | 12 | 21
1 700 | 12 | 28
1 2 | 420 | 11120
2 2 | 11 | 3
2 700 | 11 | 3
1 700 | 12 | 19
(...)
1 314 | 12 | 30
3 314 | 420 | 6
3 700 | 420 | 6
3 2 | 420 | 6
1 2 | 421 | 36
I run the job as follows, within main:
int exitCode = -1;
if ("sort".equals(args[0])) {
exitCode = ToolRunner.run(new Configuration(),
new Sorter(),
Arrays.copyOfRange(args,1,args.length));
}
Sorter extends Configured, and implements Tool as follows:
public int run(String[] args) throws Exception {
(...)
//Set job-specific params
job.setJobName("sort");
job.setInputFormatClass(SequenceFileInputFormat.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setPartitionerClass(TotalOrderPartitioner.class);
// Setup and run subset sampling
Double p = .1;
int maxNbSamples = 10;
int maxNbSplitsRead = 3;
InputSampler.Sampler<IntWritable, Text> sampler =
new InputSampler.RandomSampler<IntWritable,Text>(p,
maxNbSamples,
maxNbSplitsRead);
InputSampler.writePartitionFile(job, sampler);
// Submit the job & poll for progress until it completes
return job.waitForCompletion(true) ? 0 : -1;
}
I use the default "identity" Mapper and Reducer classes together with TotalOrderPartitioner and a RandomInputSampler to configure the partitioning. It works fine so long as I only use a single reducer task but becomes unstable when -Dmpareduce.job.reduces=N with N > 1.
By unstable, I mean that it apparently randomly
* terminates successfully;
* fails with ArrayIndexOutOfBoundsException @ InputSampler.writePartitionFile() and exits;
* or fails with IllegalArgumentException: Can't read partitions file @ TotalOrderPArtitioner.setConf() and crashes my machine.
The probability distribution of these 3 outcomes seems to vary slightly if I change the values passed to the InputSampler constructor and the number of reduce tasks, but the tool is never stable.
Can anyone shed light on this at all?