You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Dennis Kubes <nu...@dragonflymc.com> on 2006/05/15 23:55:09 UTC

File readers inside of Mappers ad Reducers

How would someone go about reading Map and Sequence file contents in 
Mappers and Reducers?  Is it best to only use the addInputDirectory 
method and find a way to get all of the data one needs by key or is 
there a good way to read file contents inside of map and reduce calls. 

I am not asking about what classes (i.e. MapFile.Reader, etc.)  I want 
to know a consistent way to do it.  Right now I am using a static 
reference in the main class, called by an inner class.  This works well 
on one machine but I don't think it is going to work across nodes.  Here 
is how I have it now:

class x {

  private static MapFile.Reader reader = null;

  public x() {
    reader = new MapFile.Reader...
    create mapreduce job
       setMapperClass(InnerX.class);
    submit job
    close reader.
  }

  class InnerX {
 
    map(WritableComparable key, Writable value, OutputCollector output, 
Reporter reporter) {
      do something with reader...
    }
  }

}

Is there a better way to do this?

Dennis