You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by lijo george <ge...@gmail.com> on 2010/09/13 06:34:51 UTC
Help in accessing intermediate results of an iterative mapreduce algorithm
Hi,
I'm trying to implement an iterative algorithm using hadoop map reduce. ( QR
decomposition using Gram Schmidt).
Presently I'm calling the algorithm inside the main() method like the
following. I'm using the output of the present iteration as the input of the
next and this is working fine.
for(counter = 0;counter < n;counter++)
{
JobConf conf = new JobConf(QRDecomposition.class);
conf.setMapperClass(MapClass.class);
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Text.class);
conf.setInputFormat(KeyValueTextInputFormat.class);
Path inputdir = new Path(args[0]+Integer.toString(counter));
Path outdir = new Path(args[2]+Integer.toString(counter+1));
FileInputFormat.setInputPaths(conf, inputdir);
FileOutputFormat.setOutputPath(conf, outdir);
conf.setNumMapTasks(Integer.parseInt(args[3]));
conf.set("iteration_count", Integer.toString(counter));
RunningJob job = JobClient.runJob(conf);
}
But I need to use another input in the algorithm which is a single column
vector. This is written by one of the map tasks depending on a condition.
if(keyReceived == (keyToUse+1))
{
sfw.append(Key,new Text(stb.toString())); // sfw is SequenceFile.Writer
}
My problem is how to write this data so that it is available in the next
iteration. I tried using sequenceFile like the above.
But the problem is creating the sequencefile writer inside the configure of
which map task.
Also how to distribute this file to the other map tasks. As this is getting
written during the program only , I guess Distributed cache is not an
option. Also
we require the unique file name for the DC.
I'm thinking of using side-effect files for this, but I think it will write
the output to the same output directory as the map/reduce. Also the filename
needs to be uniquely specified. This cannot be known exactly in the program
as we do not know which mapper did the task.
Please let me know what I need to do to solve this.