reading a directory children in DFS?

[Deyaa Adranale <de...@iais.fraunhofer.de>]

hello,
i have a problem in reading the children of a directory in the 
distributed file system of hadoop:
when I read the results of the reduce, I know the output folder (which i 
have specified using JobConf), but I don't know the file names inside 
it, and i still does not know how to access them using Java code
I have tried this:

        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);

        String outputDir = ....
        Path inDir = new Path("");
       
        if (!fs.exists(inDir))
          throw new Exception("Directory does not exist");
      
        File jInDir = new RawLocalFileSystem().pathToFile(inDir);
        String[] files = jInDir.list();       
        for (int i=0; i<files.length; i++) {
            Path inFile = new Path(inDir, files[i]);



but the files array is null, and i get a null pointer exception when 
files.length.

any suggestions? I have searched on the internet, wiki and the archive, 
but could not find something useful.


thanks for the help

Deyaa

Re: reading a directory children in DFS?

[Hairong Kuang <ha...@yahoo-inc.com>]

Your code is trying to list a directory in the local file system. You should
use the dfs handler instead.
 Path[] children =
      FileUtil.status2paths(dfs.listStatus(parentDirectoryPath));

Hairong



On 5/20/08 8:17 AM, "Deyaa Adranale" <de...@iais.fraunhofer.de>
wrote:

> hello,
> i have a problem in reading the children of a directory in the
> distributed file system of hadoop:
> when I read the results of the reduce, I know the output folder (which i
> have specified using JobConf), but I don't know the file names inside
> it, and i still does not know how to access them using Java code
> I have tried this:
> 
>         Configuration conf = new Configuration();
>         FileSystem fs = FileSystem.get(conf);
> 
>         String outputDir = ....
>         Path inDir = new Path("");
>        
>         if (!fs.exists(inDir))
>           throw new Exception("Directory does not exist");
>       
>         File jInDir = new RawLocalFileSystem().pathToFile(inDir);
>         String[] files = jInDir.list();
>         for (int i=0; i<files.length; i++) {
>             Path inFile = new Path(inDir, files[i]);
> 
> 
> 
> but the files array is null, and i get a null pointer exception when
> files.length.
> 
> any suggestions? I have searched on the internet, wiki and the archive,
> but could not find something useful.
> 
> 
> thanks for the help
> 
> Deyaa