Writing map output files to local file system

[Yair Even-Zohar <ya...@revenuescience.com>]

Can I write the output of the map processes directly to a local
files-system without the need to use copyToLocal? 

I'm trying the following but the map reduce still writes the output to
the HDFS  path

 

RawLocalFileSystem rlfs = new RawLocalFileSystem();

rlfs.setWorkingDirectory(new Path("/home/hadoop/test")); 

FileOutputFormat.setOutputPath(c, rlfs.getWorkingDirectory());

 

 

Identical to when I was using:

 

FileOutputFormat.setOutputPath(c, new Path("/home/hadoop/test"));

 

 

Thanks

-Yair

Re: Writing map output files to local file system

[stack <st...@duboce.net>]

Try using fully-qualified paths; i.e. use "file:///blah/blah" instead of 
"/blah/blah".  Is RawLocalFileSytem right?  Maybe use 
FileSystem.getLocal(Configuration).
St.Ack



Yair Even-Zohar wrote:
> Can I write the output of the map processes directly to a local
> files-system without the need to use copyToLocal? 
>
> I'm trying the following but the map reduce still writes the output to
> the HDFS  path
>
>  
>
> RawLocalFileSystem rlfs = new RawLocalFileSystem();
>
> rlfs.setWorkingDirectory(new Path("/home/hadoop/test")); 
>
> FileOutputFormat.setOutputPath(c, rlfs.getWorkingDirectory());
>
>  
>
>  
>
> Identical to when I was using:
>
>  
>
> FileOutputFormat.setOutputPath(c, new Path("/home/hadoop/test"));
>
>  
>
>  
>
> Thanks
>
> -Yair
>
>
>