You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Naama Kraus <na...@gmail.com> on 2008/03/11 08:17:17 UTC
HDFS interface
Hi,
I'd be interested in information about interfaces to HDFS other then the
DFSShell commands. I've seen threads about dfs and fuse, dfs and WebDav.
Could anyone provide more details or point me to related resources ? What's
the status of these ?
Thanks, Naama
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: HDFS interface
Posted by Eddie C <ed...@gmail.com>.
I used this code like this inside of a tomcat web application. It
works. Shared webserver filesystem :)
On Wed, Mar 12, 2008 at 4:50 PM, Hairong Kuang <ha...@yahoo-inc.com> wrote:
> http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
>
> Hairong
>
>
>
>
> On 3/12/08 1:21 PM, "Arun C Murthy" <ar...@yahoo-inc.com> wrote:
>
> >
> > http://hadoop.apache.org/core/docs/r0.16.0/hdfs_user_guide.html
> >
> > Arun
> >
> > On Mar 12, 2008, at 1:16 PM, Cagdas Gerede wrote:
> >
> >> I would like to use HDFS component of Hadoop but not interested in
> >> MapReduce.
> >> All the Hadoop examples I have seen so far uses MapReduce classes
> >> and from
> >> these examples there is no reference to HDFS classes including File
> >> System
> >> API of Hadoop (http://hadoop.apache.org/core/docs/current/api/org/
> >> apache/hadoop/fs/FileSystem.html
> >> )<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/
> >> fs/FileSystem.html>
> >> Everything seems to happen under the hood.
> >>
> >> I was wondering if there is any example source code that is using HDFS
> >> directly.
> >>
> >>
> >> Thanks,
> >>
> >> - CEG
> >
>
>
Re: HDFS interface
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
If you add the configuration directory to the class path, the configuration
files will be automatically loaded.
Hairong
On 3/12/08 5:32 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
> I found the solution. Please let me know if you have a better idea.
>
> I added the following addResource lines.
>
> Configuration conf = new Configuration();
>
> conf.addResource(new Path("location_of_hadoop-default.xml"));
> conf.addResource(new Path("location_of_hadoop-site.xml"));
>
> FileSystem fs = FileSystem.get(conf);
>
> (Would be good to update the wiki page).
>
> - CEG
>
>
> On Wed, Mar 12, 2008 at 5:04 PM, Cagdas Gerede <ca...@gmail.com>
> wrote:
>
>> I see the following paragraphs in the wiki (
>> http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample)<http://wiki.apache.o
>> rg/hadoop/HadoopDfsReadWriteExample>
>>
>>> Create a [image: [WWW]]
>>> FileSystem<http://hadoop.apache.org/core/api/org/apache/hadoop/fs/FileSystem
>>> .html>instance by passing a new Configuration object. Please note that the
>> following example code assumes that the >Configuration object will
>> automatically load the *hadoop-default.xml* and
>> *hadoop-site.xml*configuration files. You may need to explicitly add these
>> resource paths if
>> you are not running inside of the Hadoop runtime environment.
>>
>> and
>>
>>> Configuration conf = new Configuration();
>>> FileSystem fs = FileSystem.get(conf);
>>
>> When I do
>>
>> Path[] apples = fs.globPaths(new Path("*"));
>> for(Path apple : apples) {
>> System.out.println(apple);
>> }
>>
>>
>> It prints out all the local file names.
>>
>> How do I point my application to running HDFS instance?
>> What does "explicitly add these resource paths if you are not running
>> inside of the Hadoop runtime environment." mean?
>>
>> Thanks,
>>
>> - CEG
>>
>>
>>
>>
>
Re: HDFS interface
Posted by Cagdas Gerede <ca...@gmail.com>.
I found the solution. Please let me know if you have a better idea.
I added the following addResource lines.
Configuration conf = new Configuration();
conf.addResource(new Path("location_of_hadoop-default.xml"));
conf.addResource(new Path("location_of_hadoop-site.xml"));
FileSystem fs = FileSystem.get(conf);
(Would be good to update the wiki page).
- CEG
On Wed, Mar 12, 2008 at 5:04 PM, Cagdas Gerede <ca...@gmail.com>
wrote:
> I see the following paragraphs in the wiki (
> http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample)<http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample>
>
> >Create a [image: [WWW]] FileSystem<http://hadoop.apache.org/core/api/org/apache/hadoop/fs/FileSystem.html>instance by passing a new Configuration object. Please note that the
> following example code assumes that the >Configuration object will
> automatically load the *hadoop-default.xml* and *hadoop-site.xml*configuration files. You may need to explicitly add these resource paths if
> you are not running inside of the Hadoop runtime environment.
>
> and
>
> > Configuration conf = new Configuration();
> > FileSystem fs = FileSystem.get(conf);
>
> When I do
>
> Path[] apples = fs.globPaths(new Path("*"));
> for(Path apple : apples) {
> System.out.println(apple);
> }
>
>
> It prints out all the local file names.
>
> How do I point my application to running HDFS instance?
> What does "explicitly add these resource paths if you are not running
> inside of the Hadoop runtime environment." mean?
>
> Thanks,
>
> - CEG
>
>
>
>
--
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://www.cs.ucsb.edu/~gerede
Pronunciation: http://www.cs.ucsb.edu/~gerede/cagdas.html
Re: HDFS interface
Posted by Cagdas Gerede <ca...@gmail.com>.
I see the following paragraphs in the wiki (
http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample)<http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample>
>Create a [image: [WWW]]
FileSystem<http://hadoop.apache.org/core/api/org/apache/hadoop/fs/FileSystem.html>instance
by passing a new Configuration object. Please note that the
following example code assumes that the >Configuration object will
automatically load the *hadoop-default.xml* and
*hadoop-site.xml*configuration files. You may need to explicitly add
these resource paths if
you are not running inside of the Hadoop runtime environment.
and
> Configuration conf = new Configuration();
> FileSystem fs = FileSystem.get(conf);
When I do
Path[] apples = fs.globPaths(new Path("*"));
for(Path apple : apples) {
System.out.println(apple);
}
It prints out all the local file names.
How do I point my application to running HDFS instance?
What does "explicitly add these resource paths if you are not running inside
of the Hadoop runtime environment." mean?
Thanks,
- CEG
Re: HDFS interface
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
Hairong
On 3/12/08 1:21 PM, "Arun C Murthy" <ar...@yahoo-inc.com> wrote:
>
> http://hadoop.apache.org/core/docs/r0.16.0/hdfs_user_guide.html
>
> Arun
>
> On Mar 12, 2008, at 1:16 PM, Cagdas Gerede wrote:
>
>> I would like to use HDFS component of Hadoop but not interested in
>> MapReduce.
>> All the Hadoop examples I have seen so far uses MapReduce classes
>> and from
>> these examples there is no reference to HDFS classes including File
>> System
>> API of Hadoop (http://hadoop.apache.org/core/docs/current/api/org/
>> apache/hadoop/fs/FileSystem.html
>> )<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/
>> fs/FileSystem.html>
>> Everything seems to happen under the hood.
>>
>> I was wondering if there is any example source code that is using HDFS
>> directly.
>>
>>
>> Thanks,
>>
>> - CEG
>
Re: HDFS interface
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
http://hadoop.apache.org/core/docs/r0.16.0/hdfs_user_guide.html
Arun
On Mar 12, 2008, at 1:16 PM, Cagdas Gerede wrote:
> I would like to use HDFS component of Hadoop but not interested in
> MapReduce.
> All the Hadoop examples I have seen so far uses MapReduce classes
> and from
> these examples there is no reference to HDFS classes including File
> System
> API of Hadoop (http://hadoop.apache.org/core/docs/current/api/org/
> apache/hadoop/fs/FileSystem.html
> )<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/
> fs/FileSystem.html>
> Everything seems to happen under the hood.
>
> I was wondering if there is any example source code that is using HDFS
> directly.
>
>
> Thanks,
>
> - CEG
Re: HDFS interface
Posted by Cagdas Gerede <ca...@gmail.com>.
I would like to use HDFS component of Hadoop but not interested in
MapReduce.
All the Hadoop examples I have seen so far uses MapReduce classes and from
these examples there is no reference to HDFS classes including File System
API of Hadoop (http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileSystem.html
)<http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileSystem.html>
Everything seems to happen under the hood.
I was wondering if there is any example source code that is using HDFS
directly.
Thanks,
- CEG
RE: HDFS interface
Posted by dhruba Borthakur <dh...@yahoo-inc.com>.
HDFS can be accessed using the FileSystem API
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/File
System.html
The HDFS Namenode protocol can be found in
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/Nam
eNode.html
thanks,
dhruba
-----Original Message-----
From: Naama Kraus [mailto:naamakraus@gmail.com]
Sent: Tuesday, March 11, 2008 12:17 AM
To: hadoop-core
Subject: HDFS interface
Hi,
I'd be interested in information about interfaces to HDFS other then the
DFSShell commands. I've seen threads about dfs and fuse, dfs and WebDav.
Could anyone provide more details or point me to related resources ?
What's
the status of these ?
Thanks, Naama
--
oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00 oo 00
oo
00 oo 00 oo
"If you want your children to be intelligent, read them fairy tales. If
you
want them to be more intelligent, read them more fairy tales." (Albert
Einstein)
Re: HDFS interface
Posted by Pete Wyckoff <pw...@facebook.com>.
Hi Naama,
This JIRA is tracking both the fuse and webdav efforts:
https://issues.apache.org/jira/browse/HADOOP-4
-- pete
On 3/10/08 11:17 PM, "Naama Kraus" <na...@gmail.com> wrote:
> Hi,
>
> I'd be interested in information about interfaces to HDFS other then the
> DFSShell commands. I've seen threads about dfs and fuse, dfs and WebDav.
> Could anyone provide more details or point me to related resources ? What's
> the status of these ?
>
> Thanks, Naama