An FTP client for HDFS

[ANKUR GOEL <an...@corp.aol.com>]

Hi Folks,
               There is a requirement at my end to have an FTP client 
that can be used to pull data from a various remote FTP servers and load 
it into  HDFS.  Consequently I am thinking about utilizing Apache 
Commons FTP client to write a command line FTP shell that can be used to 
pull data from an FTP server and store into HDFS.

Would it be useful for the community ?
As an alternative I considered using HADOOP-4 patch but its not yet 
fully tested and committed.

Any thoughts.

Thanks
-Ankur

RE: An FTP client for HDFS

["Goel, Ankur" <An...@corp.aol.com>]

Sounds like we can share ideas over this. I have a JIRA filed for an FTP
server over HDFS
https://issues.apache.org/jira/browse/HADOOP-3199. This will provide FTP
access over HDFS.
May be you would like to take a look at it.

What I also need is an FTP client that stores content directly into
HDFS. I have already 
started work on this as it is of a greater priority for me. I am would
be developing it as 
a command line FTP shell that understands the underlying HDFS and also
connects to a remote 
FTP server.

I haven't opened a JIRA issue for it as yet, but may be you can go ahead
and open it :-)

-Ankur

-----Original Message-----
From: Nathan Fiedler [mailto:nathanfiedler@gmail.com] 
Sent: Friday, April 11, 2008 10:56 PM
To: core-dev@hadoop.apache.org
Subject: Re: An FTP client for HDFS

If you don't write it, I just might. It would eventually be of use to
me, and it would be a fun way to get involved in Hadoop. How I see using
this is to set up a small cluster to use as a reliable file store, and
use ftp to programmatically access those files. Could be useful for
deployment, such as via cfengine for Bcfg2 (that is, store the packages
and other binaries in HDFS, let the client retrieve them via ftp).

n

Re: An FTP client for HDFS

[Nathan Fiedler <na...@gmail.com>]

If you don't write it, I just might. It would eventually be of use to
me, and it would be a fun way to get involved in Hadoop. How I see
using this is to set up a small cluster to use as a reliable file
store, and use ftp to programmatically access those files. Could be
useful for deployment, such as via cfengine for Bcfg2 (that is, store
the packages and other binaries in HDFS, let the client retrieve them
via ftp).

n