You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Cam Bazz <ca...@gmail.com> on 2008/01/21 14:11:45 UTC
hadoop file system browser
hello
is there any utility to browse hadoop fs?
Best Regards,
-C.B.
Re: hadoop file system browser
Posted by Enis Soztutar <en...@gmail.com>.
Yes, you can solve the bottleneck by starting a webdav server on each
client. But this would include the burden to manage the servers etc. and
it may not be the intended use case for webdav. But we can further
discuss the architecture in the relevant issue.
Alban Chevignard wrote:
> Thanks for the clarification. I agree that running a single WebDAV
> server for all clients would make it a bottleneck. But I can't see
> anything in the current WebDAV server implementation that precludes
> running an instance of it on each client. It seems to me that would
> solve any bottleneck issue.
>
> -Alban
>
> On Jan 23, 2008 2:53 AM, Enis Soztutar <en...@gmail.com> wrote:
>
>> As you know, dfs client connects to the individual datanodes to
>> read/write data and has a minimal interaction with the Namenode, which
>> improves the io rate linearly(theoretically 1:1). However current
>> implementation of webdav interface, is just a server working on a single
>> machine, which translates the webdav requests to namenode. Thus the
>> whole traffic passes through this webdav server, which makes it a
>> bottleneck. I was planning to integrate webdav server with
>> namenode/datanode, and forward the requests to the other datanodes, so
>> that we can do io in parallel, but my focus on webdav has faded for now.
>>
>>
>>
>>
>> Alban Chevignard wrote:
>>
>>> What are the scalability issues associated with the current WebDAV interface?
>>>
>>> Thanks,
>>> -Alban
>>>
>>> On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
>>>
>>>
>>>> Webdav interface for hadoop works as it is, but it needs a major
>>>> redesign to be scalable, however it is still useful. It has even been
>>>> used with windows explorer defining the webdav server as a remote service.
>>>>
>>>>
>>>> Ted Dunning wrote:
>>>>
>>>>
>>>>> There has been significant work on building a web-DAV interface for HDFS. I
>>>>> haven't heard any news for some time, however.
>>>>>
>>>>>
>>>>> On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>> The Eclipse plug-in also features a DFS browser.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Yep. That's all true, I don't mean to self-promote, because there really isn't
>>>>>> that much to advertise ;) I was just quite attached to file manager-like user
>>>>>> interface; the mucommander clone I posted served me as a browser, but also for
>>>>>> rudimentary file operations (copying to/from, deleting folders etc.). In my
>>>>>> experience it's been quite handy.
>>>>>>
>>>>>> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
>>>>>> so
>>>>>> that HDFS filesystem is transparent to use for other apps.
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>
>
Re: hadoop file system browser
Posted by Alban Chevignard <al...@gmail.com>.
Thanks for the clarification. I agree that running a single WebDAV
server for all clients would make it a bottleneck. But I can't see
anything in the current WebDAV server implementation that precludes
running an instance of it on each client. It seems to me that would
solve any bottleneck issue.
-Alban
On Jan 23, 2008 2:53 AM, Enis Soztutar <en...@gmail.com> wrote:
> As you know, dfs client connects to the individual datanodes to
> read/write data and has a minimal interaction with the Namenode, which
> improves the io rate linearly(theoretically 1:1). However current
> implementation of webdav interface, is just a server working on a single
> machine, which translates the webdav requests to namenode. Thus the
> whole traffic passes through this webdav server, which makes it a
> bottleneck. I was planning to integrate webdav server with
> namenode/datanode, and forward the requests to the other datanodes, so
> that we can do io in parallel, but my focus on webdav has faded for now.
>
>
>
>
> Alban Chevignard wrote:
> > What are the scalability issues associated with the current WebDAV interface?
> >
> > Thanks,
> > -Alban
> >
> > On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
> >
> >> Webdav interface for hadoop works as it is, but it needs a major
> >> redesign to be scalable, however it is still useful. It has even been
> >> used with windows explorer defining the webdav server as a remote service.
> >>
> >>
> >> Ted Dunning wrote:
> >>
> >>> There has been significant work on building a web-DAV interface for HDFS. I
> >>> haven't heard any news for some time, however.
> >>>
> >>>
> >>> On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
> >>>
> >>>
> >>>
> >>>>> The Eclipse plug-in also features a DFS browser.
> >>>>>
> >>>>>
> >>>> Yep. That's all true, I don't mean to self-promote, because there really isn't
> >>>> that much to advertise ;) I was just quite attached to file manager-like user
> >>>> interface; the mucommander clone I posted served me as a browser, but also for
> >>>> rudimentary file operations (copying to/from, deleting folders etc.). In my
> >>>> experience it's been quite handy.
> >>>>
> >>>> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
> >>>> so
> >>>> that HDFS filesystem is transparent to use for other apps.
> >>>>
> >>>> Dawid
> >>>>
> >>>>
> >>>
> >>>
> >
> >
>
Re: hadoop file system browser
Posted by Doug Cutting <cu...@apache.org>.
Ted Dunning wrote:
> Wouldn't it work to just load balance a small farm of webdav servers? That
> would be better from a security point of view as well.
I think one should just run the webdav proxy locally whereever one needs
webdav-based access, then always connect to localhost. The namenode
host and port could be the first component of the path that's exposed to
webdav clients.
Doug
Re: hadoop file system browser
Posted by Ted Dunning <td...@veoh.com>.
Wouldn't it work to just load balance a small farm of webdav servers? That
would be better from a security point of view as well.
On 1/22/08 11:53 PM, "Enis Soztutar" <en...@gmail.com> wrote:
> As you know, dfs client connects to the individual datanodes to
> read/write data and has a minimal interaction with the Namenode, which
> improves the io rate linearly(theoretically 1:1). However current
> implementation of webdav interface, is just a server working on a single
> machine, which translates the webdav requests to namenode. Thus the
> whole traffic passes through this webdav server, which makes it a
> bottleneck. I was planning to integrate webdav server with
> namenode/datanode, and forward the requests to the other datanodes, so
> that we can do io in parallel, but my focus on webdav has faded for now.
>
>
>
> Alban Chevignard wrote:
>> What are the scalability issues associated with the current WebDAV interface?
>>
>> Thanks,
>> -Alban
>>
>> On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
>>
>>> Webdav interface for hadoop works as it is, but it needs a major
>>> redesign to be scalable, however it is still useful. It has even been
>>> used with windows explorer defining the webdav server as a remote service.
>>>
>>>
>>> Ted Dunning wrote:
>>>
>>>> There has been significant work on building a web-DAV interface for HDFS.
>>>> I
>>>> haven't heard any news for some time, however.
>>>>
>>>>
>>>> On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
>>>>
>>>>
>>>>
>>>>>> The Eclipse plug-in also features a DFS browser.
>>>>>>
>>>>>>
>>>>> Yep. That's all true, I don't mean to self-promote, because there really
>>>>> isn't
>>>>> that much to advertise ;) I was just quite attached to file manager-like
>>>>> user
>>>>> interface; the mucommander clone I posted served me as a browser, but also
>>>>> for
>>>>> rudimentary file operations (copying to/from, deleting folders etc.). In
>>>>> my
>>>>> experience it's been quite handy.
>>>>>
>>>>> It would be probably a good idea to implement a commons-vfs plugin for
>>>>> Hadoop
>>>>> so
>>>>> that HDFS filesystem is transparent to use for other apps.
>>>>>
>>>>> Dawid
>>>>>
>>>>>
>>>>
>>>>
>>
>>
Re: hadoop file system browser
Posted by Enis Soztutar <en...@gmail.com>.
As you know, dfs client connects to the individual datanodes to
read/write data and has a minimal interaction with the Namenode, which
improves the io rate linearly(theoretically 1:1). However current
implementation of webdav interface, is just a server working on a single
machine, which translates the webdav requests to namenode. Thus the
whole traffic passes through this webdav server, which makes it a
bottleneck. I was planning to integrate webdav server with
namenode/datanode, and forward the requests to the other datanodes, so
that we can do io in parallel, but my focus on webdav has faded for now.
Alban Chevignard wrote:
> What are the scalability issues associated with the current WebDAV interface?
>
> Thanks,
> -Alban
>
> On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
>
>> Webdav interface for hadoop works as it is, but it needs a major
>> redesign to be scalable, however it is still useful. It has even been
>> used with windows explorer defining the webdav server as a remote service.
>>
>>
>> Ted Dunning wrote:
>>
>>> There has been significant work on building a web-DAV interface for HDFS. I
>>> haven't heard any news for some time, however.
>>>
>>>
>>> On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
>>>
>>>
>>>
>>>>> The Eclipse plug-in also features a DFS browser.
>>>>>
>>>>>
>>>> Yep. That's all true, I don't mean to self-promote, because there really isn't
>>>> that much to advertise ;) I was just quite attached to file manager-like user
>>>> interface; the mucommander clone I posted served me as a browser, but also for
>>>> rudimentary file operations (copying to/from, deleting folders etc.). In my
>>>> experience it's been quite handy.
>>>>
>>>> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
>>>> so
>>>> that HDFS filesystem is transparent to use for other apps.
>>>>
>>>> Dawid
>>>>
>>>>
>>>
>>>
>
>
Re: hadoop file system browser
Posted by Jason Venner <ja...@attributor.com>.
With very minor changes it works with 0.15.2, read only.
Vetle Roeim wrote:
> On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
> <je...@gmail.com> wrote:
>
>> we use FUSE: who wants a gui when you could have a shell?
>> http://issues.apache.org/jira/browse/HADOOP-4
>
> Does this work with newer versions of Hadoop?
>
>
> [...]
--
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested
Re: hadoop file system browser
Posted by Pete Wyckoff <pw...@facebook.com>.
Another note is we implemented the trash feature in fuse. This could be
turned on and off with ioctl. We also don¹t allow removal of certain
directories which again can be configured with ioctl. (but isn¹t yet)
-- pete
On 1/24/08 10:39 AM, "Vetle Roeim" <ve...@opera.com> wrote:
> On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
> <je...@gmail.com> wrote:
>
>> > we use FUSE: who wants a gui when you could have a shell?
>> > http://issues.apache.org/jira/browse/HADOOP-4
>
> Does this work with newer versions of Hadoop?
>
>
> [...]
> --
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>
Re: hadoop file system browser
Posted by Jason Venner <ja...@attributor.com>.
The only change needed for 0.15.2 was to change the references to
info[0].mCreationTime into references to info[0].mLastMod
Pete Wyckoff wrote:
> Right now its tested with 0.14.4. It also includes rmdir, rm, mkdir, mv.
> I¹ve implemented write, but it has to wait for appends to work in Hadoop
> because of the Fuse protocol.
>
> Our strategy thus far has been to use FUSE on a single box and then NFS
> export it to other machines. We don¹t do heavy, heavy operations on it, so
> it isn¹t a performance problem. The things I think are most useful anyway
> are ls, find, du, mkdir, rmdir, rm and mv none of which tax FUSe much.
>
> -- pete
>
>
> On 1/24/08 10:39 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>
>
>> On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
>> <je...@gmail.com> wrote:
>>
>>
>>>> we use FUSE: who wants a gui when you could have a shell?
>>>> http://issues.apache.org/jira/browse/HADOOP-4
>>>>
>> Does this work with newer versions of Hadoop?
>>
>>
>> [...]
>> --
>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>
>>
>
>
>
>
--
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested
Re: hadoop file system browser
Posted by Vetle Roeim <ve...@opera.com>.
Yes, please post it again. :) Lack of trash and directory protection
shouldn't be an issue for my needs.
On Thu, 24 Jan 2008 20:11:26 +0100, Pete Wyckoff <pw...@facebook.com>
wrote:
>
> I can post it again, but it doesn¹t include ioctl commands so the trash
> feature cannot be configured. I can still create a flag and default it to
> false. And also the directory protection isn¹t configurable so I can set
> a
> flag to false. The main directory we protect here is /user/facebook for
> data
> (and job :) ) protection purposes.
>
>
> On 1/24/08 10:55 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>
>>> >
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: hadoop file system browser
Posted by Pete Wyckoff <pw...@facebook.com>.
I can post it again, but it doesn¹t include ioctl commands so the trash
feature cannot be configured. I can still create a flag and default it to
false. And also the directory protection isn¹t configurable so I can set a
flag to false. The main directory we protect here is /user/facebook for data
(and job :) ) protection purposes.
On 1/24/08 10:55 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>> >
Re: hadoop file system browser
Posted by Vetle Roeim <ve...@opera.com>.
Thanks!
On Thu, 24 Jan 2008 20:29:20 +0100, Pete Wyckoff <pw...@facebook.com>
wrote:
>
> I attached the newest version to:
> https://issues.apache.org/jira/browse/HADOOP-4
>
> Still a work in progress and any help appreciated. Not much by way of
> instructions but here are some:
>
> 1. download and install fuse and do a modprobe fuse
> 2. modify fuse_dfs.c¹s Makefile to have the right paths for fuse, hdfs.h
> and
> jni
> 3. ensure you have hadoop in your class path and the jni stuff in your
> library path
> 4. mkdir /tmp/hdfs
> 5. ./fuse_dfs dfs://hadoop_namenode:9000 /tmp/hdfs d
>
> Probably will be missing things in your class path and LD_LIBRARY_PATH
> when
> you do 5, so just add them and iterate. To run this as production
> quality,
> you basically need fuse_dfs in root¹s path and add a line to /etc/fstab.
> For
> people interested, I can give you my config line.
>
> -- pete
>
>
> On 1/24/08 10:55 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>
>>> >
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: hadoop file system browser
Posted by Pete Wyckoff <pw...@facebook.com>.
I attached the newest version to:
https://issues.apache.org/jira/browse/HADOOP-4
Still a work in progress and any help appreciated. Not much by way of
instructions but here are some:
1. download and install fuse and do a modprobe fuse
2. modify fuse_dfs.c¹s Makefile to have the right paths for fuse, hdfs.h and
jni
3. ensure you have hadoop in your class path and the jni stuff in your
library path
4. mkdir /tmp/hdfs
5. ./fuse_dfs dfs://hadoop_namenode:9000 /tmp/hdfs d
Probably will be missing things in your class path and LD_LIBRARY_PATH when
you do 5, so just add them and iterate. To run this as production quality,
you basically need fuse_dfs in root¹s path and add a line to /etc/fstab. For
people interested, I can give you my config line.
-- pete
On 1/24/08 10:55 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>> >
Re: hadoop file system browser
Posted by Vetle Roeim <ve...@opera.com>.
Great! Where can I get it? :)
On Thu, 24 Jan 2008 19:48:57 +0100, Pete Wyckoff <pw...@facebook.com>
wrote:
>
> Right now its tested with 0.14.4. It also includes rmdir, rm, mkdir, mv.
> I¹ve implemented write, but it has to wait for appends to work in Hadoop
> because of the Fuse protocol.
>
> Our strategy thus far has been to use FUSE on a single box and then NFS
> export it to other machines. We don¹t do heavy, heavy operations on it,
> so
> it isn¹t a performance problem. The things I think are most useful
> anyway
> are ls, find, du, mkdir, rmdir, rm and mv none of which tax FUSe much.
>
> -- pete
>
>
> On 1/24/08 10:39 AM, "Vetle Roeim" <ve...@opera.com> wrote:
>
>> On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
>> <je...@gmail.com> wrote:
>>
>>> > we use FUSE: who wants a gui when you could have a shell?
>>> > http://issues.apache.org/jira/browse/HADOOP-4
>>
>> Does this work with newer versions of Hadoop?
>>
>>
>> [...]
>> --
>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: hadoop file system browser
Posted by Pete Wyckoff <pw...@facebook.com>.
Right now its tested with 0.14.4. It also includes rmdir, rm, mkdir, mv.
I¹ve implemented write, but it has to wait for appends to work in Hadoop
because of the Fuse protocol.
Our strategy thus far has been to use FUSE on a single box and then NFS
export it to other machines. We don¹t do heavy, heavy operations on it, so
it isn¹t a performance problem. The things I think are most useful anyway
are ls, find, du, mkdir, rmdir, rm and mv none of which tax FUSe much.
-- pete
On 1/24/08 10:39 AM, "Vetle Roeim" <ve...@opera.com> wrote:
> On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
> <je...@gmail.com> wrote:
>
>> > we use FUSE: who wants a gui when you could have a shell?
>> > http://issues.apache.org/jira/browse/HADOOP-4
>
> Does this work with newer versions of Hadoop?
>
>
> [...]
> --
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>
Re: hadoop file system browser
Posted by Vetle Roeim <ve...@opera.com>.
On Tue, 22 Jan 2008 22:03:03 +0100, Jeff Hammerbacher
<je...@gmail.com> wrote:
> we use FUSE: who wants a gui when you could have a shell?
> http://issues.apache.org/jira/browse/HADOOP-4
Does this work with newer versions of Hadoop?
[...]
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: hadoop file system browser
Posted by Jeff Hammerbacher <je...@gmail.com>.
we use FUSE: who wants a gui when you could have a shell?
http://issues.apache.org/jira/browse/HADOOP-4
On Jan 22, 2008 6:50 AM, Alban Chevignard <al...@gmail.com> wrote:
> What are the scalability issues associated with the current WebDAV
> interface?
>
> Thanks,
> -Alban
>
> On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
> > Webdav interface for hadoop works as it is, but it needs a major
> > redesign to be scalable, however it is still useful. It has even been
> > used with windows explorer defining the webdav server as a remote
> service.
> >
> >
> > Ted Dunning wrote:
> > > There has been significant work on building a web-DAV interface for
> HDFS. I
> > > haven't heard any news for some time, however.
> > >
> > >
> > > On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl>
> wrote:
> > >
> > >
> > >>> The Eclipse plug-in also features a DFS browser.
> > >>>
> > >> Yep. That's all true, I don't mean to self-promote, because there
> really isn't
> > >> that much to advertise ;) I was just quite attached to file
> manager-like user
> > >> interface; the mucommander clone I posted served me as a browser, but
> also for
> > >> rudimentary file operations (copying to/from, deleting folders etc.).
> In my
> > >> experience it's been quite handy.
> > >>
> > >> It would be probably a good idea to implement a commons-vfs plugin
> for Hadoop
> > >> so
> > >> that HDFS filesystem is transparent to use for other apps.
> > >>
> > >> Dawid
> > >>
> > >
> > >
> > >
> >
>
Re: hadoop file system browser
Posted by Alban Chevignard <al...@gmail.com>.
What are the scalability issues associated with the current WebDAV interface?
Thanks,
-Alban
On Jan 22, 2008 7:27 AM, Enis Soztutar <en...@gmail.com> wrote:
> Webdav interface for hadoop works as it is, but it needs a major
> redesign to be scalable, however it is still useful. It has even been
> used with windows explorer defining the webdav server as a remote service.
>
>
> Ted Dunning wrote:
> > There has been significant work on building a web-DAV interface for HDFS. I
> > haven't heard any news for some time, however.
> >
> >
> > On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
> >
> >
> >>> The Eclipse plug-in also features a DFS browser.
> >>>
> >> Yep. That's all true, I don't mean to self-promote, because there really isn't
> >> that much to advertise ;) I was just quite attached to file manager-like user
> >> interface; the mucommander clone I posted served me as a browser, but also for
> >> rudimentary file operations (copying to/from, deleting folders etc.). In my
> >> experience it's been quite handy.
> >>
> >> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
> >> so
> >> that HDFS filesystem is transparent to use for other apps.
> >>
> >> Dawid
> >>
> >
> >
> >
>
Re: hadoop file system browser
Posted by Enis Soztutar <en...@gmail.com>.
Webdav interface for hadoop works as it is, but it needs a major
redesign to be scalable, however it is still useful. It has even been
used with windows explorer defining the webdav server as a remote service.
Ted Dunning wrote:
> There has been significant work on building a web-DAV interface for HDFS. I
> haven't heard any news for some time, however.
>
>
> On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
>
>
>>> The Eclipse plug-in also features a DFS browser.
>>>
>> Yep. That's all true, I don't mean to self-promote, because there really isn't
>> that much to advertise ;) I was just quite attached to file manager-like user
>> interface; the mucommander clone I posted served me as a browser, but also for
>> rudimentary file operations (copying to/from, deleting folders etc.). In my
>> experience it's been quite handy.
>>
>> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
>> so
>> that HDFS filesystem is transparent to use for other apps.
>>
>> Dawid
>>
>
>
>
Re: hadoop file system browser
Posted by Ted Dunning <td...@veoh.com>.
There has been significant work on building a web-DAV interface for HDFS. I
haven't heard any news for some time, however.
On 1/21/08 11:32 AM, "Dawid Weiss" <da...@cs.put.poznan.pl> wrote:
>
>> The Eclipse plug-in also features a DFS browser.
>
> Yep. That's all true, I don't mean to self-promote, because there really isn't
> that much to advertise ;) I was just quite attached to file manager-like user
> interface; the mucommander clone I posted served me as a browser, but also for
> rudimentary file operations (copying to/from, deleting folders etc.). In my
> experience it's been quite handy.
>
> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
> so
> that HDFS filesystem is transparent to use for other apps.
>
> Dawid
Re: hadoop file system browser
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> The Eclipse plug-in also features a DFS browser.
Yep. That's all true, I don't mean to self-promote, because there really isn't
that much to advertise ;) I was just quite attached to file manager-like user
interface; the mucommander clone I posted served me as a browser, but also for
rudimentary file operations (copying to/from, deleting folders etc.). In my
experience it's been quite handy.
It would be probably a good idea to implement a commons-vfs plugin for Hadoop so
that HDFS filesystem is transparent to use for other apps.
Dawid
Re: hadoop file system browser
Posted by Christophe Taton <ch...@gmail.com>.
The Eclipse plug-in also features a DFS browser.
Johannes Zillmann wrote:
> Also the dfs should be browsable over web-ui
> http://<namenode>:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>
>
> Johannes
>
> Dawid Weiss wrote:
>>
>> I posted something like this a while ago. See here.
>>
>> https://issues.apache.org/jira/browse/HADOOP-2534
>>
>> Works well for me, read the docs -- you'll need to update JARs (and
>> re-sign them) if you work with the HEAD.
>>
>> Dawid
>>
>
>
Re: hadoop file system browser
Posted by Johannes Zillmann <jz...@101tec.com>.
Also the dfs should be browsable over web-ui
http://<namenode>:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
Johannes
Dawid Weiss wrote:
>
> I posted something like this a while ago. See here.
>
> https://issues.apache.org/jira/browse/HADOOP-2534
>
> Works well for me, read the docs -- you'll need to update JARs (and
> re-sign them) if you work with the HEAD.
>
> Dawid
>
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec GmbH
Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com
Re: hadoop file system browser
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
I posted something like this a while ago. See here.
https://issues.apache.org/jira/browse/HADOOP-2534
Works well for me, read the docs -- you'll need to update JARs (and re-sign
them) if you work with the HEAD.
Dawid
Re: hadoop file system browser
Posted by Cam Bazz <ca...@gmail.com>.
Hello,
I meant like a gui. I found out this is possible tru the web interface
however.
thanks.
On Jan 21, 2008 3:17 PM, Miles Osborne <mi...@inf.ed.ac.uk> wrote:
> hadoop dfs -ls DIR
>
> (etc) allows you to see the file system
>
> Miles
>
> On 21/01/2008, Cam Bazz <ca...@gmail.com> wrote:
> >
> > hello
> > is there any utility to browse hadoop fs?
> > Best Regards,
> > -C.B.
> >
>
Re: hadoop file system browser
Posted by Miles Osborne <mi...@inf.ed.ac.uk>.
hadoop dfs -ls DIR
(etc) allows you to see the file system
Miles
On 21/01/2008, Cam Bazz <ca...@gmail.com> wrote:
>
> hello
> is there any utility to browse hadoop fs?
> Best Regards,
> -C.B.
>