Challenges using FTPClient with Hadoop distcp

[Ken Krugler <kk...@transpac.com>]

Hi all,

We're trying to use the (pretty much never used) FTPFileSystem in Hadoop, to send data from a Hadoop cluster to an FTP server.

The first challenge is that the FTPFileSystem configures FTPClient to run in active mode, which didn't work with our FTP server/firewall configuration.

So we created a PassiveFTPFileSystem that uses passive mode.

This is able to connect to the FTP server, and is able to send some files - but ultimately this copying always fails.

On the server side we see nothing in the logs (it's using vsftp 2.2.2), even with debug logging enabled.

On the Hadoop (client) side, we see a mix of errors in the logs. Most look like...

org.apache.commons.net.ftp.FTPConnectionClosedException: Connection closed without indication.
	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:267)
	at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:460)
	at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:520)
	at org.apache.commons.net.ftp.FTP.cwd(FTP.java:745)

I'm wondering if there's any issue running 14 parallel FTPClient sessions from a single server - e.g. collisions in port numbers, though from my reading of the code that doesn't seem possible.

Thanks for any input.

Regards,

-- Ken

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr