[Lucene-hadoop Wiki] Update of "LibHDFS" by Arun C Murthy

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by Arun C Murthy:
http://wiki.apache.org/lucene-hadoop/LibHDFS

The comment on the change is:
First cut

New page:
= libhdfs =

 ''''libhdfs'''' is a [http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/jniTOC.html JNI] based C api for Hadoop's DFS. 
 It provides a simple subset of C apis to manipulate DFS files and the filesystem. libhdfs is available for [http://www.apache.org/dyn/closer.cgi/lucene/hadoop/ download] as a part of Hadoop itself. The source for libhdfs is available for browsing [http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/c%2B%2B/libhdfs/ here].


==== Table Of Contents ====
 1. [#Overview Overview]
 2. [#Setup Setup]
 3. [#Apis APIs]
  a. [#FileSystemApis FileSystem Manipulation]
  b. [#FileApis File Manipulation]
  c. [#Threading Support for Threads]
 4. [#Contact Assistance/Bugs]
 5. [#Conclusion Conclusion]


[[Anchor(Overview)]]
===== Overview =====
 [http://wiki.apache.org/lucene-hadoop/DFS Here] is an overview of Hadoop's DFS. The javadocs for DFS are available [http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/fs/package-summary.html here] and [http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/dfs/package-summary.html here]. '''libdhfs''' is a simple JNI based C api for accessing and manipulating Hadoop's DFS from native code. It offers a simple subset of the same functionality.

[[Anchor(Setup)]]
===== Setup =====
 It is necessary to setup Hadoop's DFS itself first. The information to setup Hadoop is available [http://lucene.apache.org/hadoop/docs/api/overview-summary.html#overview_description here]. Once you have a working setup, you will need to get into the '''src/c++/libhdfs''' directory and use the Makefile to build libhdfs (in case of issues use [#Contact this]). Once you have successfully built libhdfs you can link it into your programs and are good to go.

[[Anchor(Apis)]]
===== APIs =====
 This section describes the various apis provided by '''libhdfs''' to manipulate the DFS. It is classified into apis which manipulate individual files and those which manipulate the filesystem itself. (Please see the doxygen documentation [# here] for details of individual apis.)

[[Anchor(FileSystemApis)]]
''!FileSystem APIs''
[[BR]]
 libhdfs provides apis for both '''generic''' manipulation of the filesytem (create directories, copy/move files etc.) and also some very DFS specific functionality (get information on file replication etc.).
[[BR]]
 At startup one should use the ''hdfsConnect'' api to connect to the DFS before any operations can be performed (on files or the filesystem); the analogous ''hdfsDisconnect'' performs a clean teardown of the connection.
[[BR]]
 Generic operations:
 * hdfsCopy (across filesytems also)
 * hdfsMove (across filesytems also)
 * hdfsRename
 * hdfsDelete
[[BR]]
 libhdfs also provides apis for manipulating directories on the DFS:
 * hdfsCreateDirectory
 * hdfsSetWorkingDirectory
 * hdfsGetWorkingDirectory
 * hdfsListDirectory / hdfsGetPathInfo / hdfsFreeFileInfo
[[BR]]
 The apis to query the filesytems for various properties:
 * hdfsGetHosts
 * hdfsGetDefaultBlockSize
 * hdfsGetUsed / hdfsGetCapacity
 
[[BR]]
[[Anchor(FileApis)]]
''File APIs''
[[BR]]
 libhdfs provides posix-like apis to manipulate individual files (create, read/write, query etc.) listed below:
 * hdfsOpenFile / hdfsCloseFile
 * hdfsRead / hdfsWrite
 * hdfsTell / hdfsSeek
 * hdfsFlush
 * hdfsAvailable
 * hdfsLock / hdfsUnlock

[[BR]]
[[Anchor(Threading)]]
''Using libhdfs in Threaded Applications''
[[BR]]
 libhdfs can be used in threaded applications using the !Posix !Threads. However to carefully interact with JNI's global/local references the user has to explicitly call the ''hdfsConvertToGlobalRef'' / ''hdfsDeleteGlobalRef'' apis.

[[BR]]
[[Anchor(Contact)]]
===== Contact Information =====

 Please drop us an email at '''hadoop-users@lucene.apache.org''' if you have any questions or any suggestions. Use [http://issues.apache.org/jira/browse/HADOOP Jira]] to report bugs.

[[BR]]
[[Anchor(Conclusion)]]
===== Conclusion =====

 Thank you for your interest in Hadoop and libhdfs!