Re: How to enumerate files in the directories?

[path2727 <pa...@gmail.com>]

I think this might be a better answer to your question.  I took a lot of the
code out of the web interface they made. 

$HADOOP_HOME/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java

and 

$HADOOP_HOME/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DatanodeJspHelper.java


import java.io.File;
import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.security.PrivilegedExceptionAction;
import java.util.Date;
import java.util.List;

import org.apache.hadoop.hdfs.DFSClient;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
import org.apache.hadoop.hdfs.protocol.DirectoryListing;

public class FileTest {


   /**
    *
    * @param server the name of the namenode server. 'namenode.example.com'
    * @param port the port of the name node.
    *   mine is 54310 right now.
    * this is the info port not the other port that the slaves connect to.
    * @param dir the directory you wish to enumerate. I used '/' in this
example.
    *
    */
   public FileTest( String server, String port, String dir ) {

      String tDir = validatePath( dir );

      int namenodePort = Integer.parseInt(port);

      if( tDir != null ) {
         Configuration conf = new Configuration(true);

         UserGroupInformation ugi = null;

         try {
            ugi = UserGroupInformation.getCurrentUser();
         } catch ( IOException ioe ) { ioe.printStackTrace(); }


         InetSocketAddress inet = new InetSocketAddress( server,
namenodePort );



         if( ugi != null && inet != null && conf != null ) {
            try {
               DFSClient dfs = getDFSClient(
                   ugi, inet, conf);

               String target = dir;
               final HdfsFileStatus targetStatus = dfs.getFileInfo(target);

               if( targetStatus.isDir() ) {
                  //System.out.println("it is a directory");
                  DirectoryListing thisListing =
                     dfs.listPaths(target, HdfsFileStatus.EMPTY_NAME);
                  if (thisListing == null ||
thisListing.getPartialListing().length == 0) {
                     System.out.println("Empty directory");
                  } else {
                     //System.out.println("directory not empty");
                     HdfsFileStatus[] files =
thisListing.getPartialListing();
                     for (int i = 0; i < files.length; i++) {
                         if( files[i].isDir() ) {
                            System.out.println(" dir " +
files[i].getLocalName() );
                         } else {
                            System.out.println(" file " +
files[i].getLocalName() + files[i].getReplication()+files[i].getBlockSize()
);
                         }
                     }
                  }
               } else {
                  System.out.println("it is not a directory");
               }
            } catch ( Exception e ) { // Could be IOException or
InterruptedException
               e.printStackTrace();
            }
         } else {
           System.out.println("a requirement is null");
         }
      }
   }


   private static DFSClient getDFSClient(final UserGroupInformation user,
                                        final InetSocketAddress addr,
                                        final Configuration conf
                                        ) throws IOException,
                                                 InterruptedException {
       return
         user.doAs(new PrivilegedExceptionAction<DFSClient>() {
            public DFSClient run() throws IOException {
               return new DFSClient(addr, conf);
            }
         });
  }


   public static String validatePath(String p) {
      return p == null || p.length() == 0?
         null: new Path(p).toUri().getPath();
   }


   public static void main( String[] args ) {
      if( args.length == 3 && args[2].contains("/") && args[0].contains(".")
) {
         FileTest ft = new FileTest( args[0], args[1], args[2] );
      } else {
         System.out.println("Usage: java FileTest <serverName>
<nameNodeInfoPort> <dir> ");
         System.out.println(
       "a valid dir must have '/' in the string somewhere");
         System.out.println("a valid server must have '.' in the string
somewhere");

      }
   }

}

-- 
View this message in context: http://lucene.472066.n3.nabble.com/How-to-enumerate-files-in-the-directories-tp1325738p1685723.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Re: How to enumerate files in the directories?

[path2727 <pa...@gmail.com>]

            Configuration conf = new Configuration(true);
            conf.set( "fs.default.name", "hdfs://<namenode>:<port>");

I noticed that simply adding this line to a few of the previous posts solved
my problems.  I was frustrated because I was trying to use their examples
and it only printed my LOCAL file system.  I was using the examples as java
<program_name> and not via the '$HADOOP_HOME/bin/hadoop jar'. Since i was
trying to execute them in this way, my program wasn't loading the
Configuration object correctly.


Just thought I would add that here since it caused me frustration.
-- 
View this message in context: http://lucene.472066.n3.nabble.com/How-to-enumerate-files-in-the-directories-tp1325738p1686040.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.