You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Dennis Kubes <nu...@dragonflymc.com> on 2006/05/15 23:55:09 UTC
File readers inside of Mappers ad Reducers
How would someone go about reading Map and Sequence file contents in
Mappers and Reducers? Is it best to only use the addInputDirectory
method and find a way to get all of the data one needs by key or is
there a good way to read file contents inside of map and reduce calls.
I am not asking about what classes (i.e. MapFile.Reader, etc.) I want
to know a consistent way to do it. Right now I am using a static
reference in the main class, called by an inner class. This works well
on one machine but I don't think it is going to work across nodes. Here
is how I have it now:
class x {
private static MapFile.Reader reader = null;
public x() {
reader = new MapFile.Reader...
create mapreduce job
setMapperClass(InnerX.class);
submit job
close reader.
}
class InnerX {
map(WritableComparable key, Writable value, OutputCollector output,
Reporter reporter) {
do something with reader...
}
}
}
Is there a better way to do this?
Dennis