Re: FTP transport alternatives for faster throughput

[DevNull43 <de...@gmail.com>]

Yes throughput is an issue, and I always try to get the best performance
out of anything I have or build.

I tend to fine tune my environments with the best TCP performance,
following for example this guide: http://fasterdata.es.net/tuning.html

For instance, having Linux servers with 2.6.18+ kernel with CUBIC-TCP
congestion control, increases performance for WAN transfers without the
need to modify applications.

Aside from server tunning, I try to find out the faster way to transfer
files. Splitting huge files in small pieces and doing parallel download
already does a fine job. However I was interested in other
protocols/options.

I have found this OpenSource protocol http://udt.sourceforge.net/ which
has a Java library http://sourceforge.net/projects/udt-java/ and looks
promising. I'm currently testing it.

On the other side, you get a list of features using "FEAT" command when
you log to the ftp server. Lets say you implement the UDT protocol, so
you get a "UDT" list from FEAT. 

If a client supports this feature it can use this protocol, otherwise
keeps using standard default ones. This is then not an issue.

Most commercial FTP clients tend to add new features quickly when you
provide the spec/library and this feature is spread on some servers.
This provides them a plus against competitors.

I still think having such a nice protocol in FtpServer should be a plus,
considering FTP servers are the preferred ones for data transfers.

Don't you think so? :D

Regards.


On Sun, 2010-04-25 at 12:19 +0200, David Latorre wrote:
> Good day,
> 
> Thanks a lot for the information, Dana.  I  think your information is
> pretty reasonable for a  environment such as the one  you're
> describing. Since you have an evaluation version that potential
> consumers may download and use, I guess most of your customers are
> companies that actually have a need to  improve the throughput and/or
> latency of their transfer solutions... are there any other
> 'requirements for the enterpise' you have implemented? Like an
> improved mechanism to verify the integrity of a transferred file or
> the ability to execute a script/program on file reception...?
> Is it a little too much if I ask you which of these features are  the
> most demanded (or praised) by your customers?  In our case, FTP is
> doing OK in terms of throughput but since each customer is using a
> different communications suite or FTP client (some of them rather
> limited) it is difficult to get to an agreement on how to, e.g. signal
> that a transfer was completed successfully.
> 
> 
> Devnull, is FTP throughtput really an issue for you?  It is not
> impossible to "design and develop" our own alternative to TCP  (not
> that it sounds easy...) but I wouldn't spend my time on this. Of
> course TCP can be improved in terms of efficiency, but  that would
> mean , among other things, that you are no longer compatible with any
> FTP client out there!
> 
> 
> 
> 
> 
> 2010/4/23 Dana Merk <dm...@dataexpedition.com>:
> > DevNull43,
> >
> > I appreciate the quick and detailed reply.  Sounds like your tests yielded
> > about what we'd expect, given the factors in play.  Generally, on an
> > unstressed network, LAN-scenarios, we expect ExpeDat and FTP to come within
> > a few % points of each other in terms of performance.
> >
> > As you noted, our protocol is really designed for more of the enterprise WAN
> > space, where we typically see very big data sets going across larger data
> > paths typically wrought with high congestion, latency, and packet loss.  In
> > those cases, there is usually no comparison to plain FTP.  In essence,
> > MTP/IP is going to strive to fill the pipe as efficiently and quickly as
> > possible.  So, scenarios where TCP does a fine job of that (LAN, short-hops,
> > etc.), really don't tend to be great use scenarios for us.
> >
> > Of course, if we can ever provide any assistance from this angle / line of
> > thinking, please do not hesitate to keep us in mind and contact us.
> >
> > Thank you again for the reply.
> >
> > Have a nice weekend,
> > Dana
> >
> >
> > On Apr 23, 2010, at 3:06 PM, DevNull43 wrote:
> >
> >> Hi Dana,
> >>
> >> Thanks a lot for your email, as you known FtpServer is a OpenSource
> >> project from the apache foundation, that I'm using in a simple
> >> application for personal use.
> >>
> >> The thread I opened was more focused to implement such high-speed
> >> protocols in the OpenSource community, and asked feedback if anybody
> >> used them, or why none current OpenSource project tried to implement
> >> it's own.
> >>
> >> I did download ExpeDat for evaluating if your claims are accurate. I
> >> have to say I only tested ExpeDat and none of the other competitors due
> >> the facility to download the evaluation, and also because I'm linux
> >> based and your software fits here, so congrats on that side.
> >>
> >> Having this said, the test showed less performance than using plain FTP
> >> connection ( without any kind of SSL ), on a Wireless 802.11g LAN. I
> >> transferred a 700M Ubuntu ISO file a 'bit' faster using plain FTP ( 15
> >> seconds faster).
> >>
> >> I repetead the test from my ADSL line (12Mbit) , having the remote
> >> server on a 100Mbit link located in datacenter, with 54 ms latency, and
> >> FTP file transfered 10 seconds faster as well.
> >>
> >> I made probably simple tests, or maybe your protocol has faster
> >> throughput for longer distance WAN with higher latency, more congestion
> >> networks or etc.
> >>
> >> Nevertheless thanks a lot for your email.
> >>
> >> Best regards,
> >>
> >> DevNull43.
> >>
> >> On Fri, 2010-04-23 at 14:32 -0400, Dana Merk wrote:
> >>>
> >>> "Dev Null",
> >>>
> >>> I apologize for the "cold" email, but I wanted to follow-up to a
> >>> posting I just came across that you posted to ftpserver-users
> >>> regarding FTP alternatives.  Our company, Data Expedition, Inc., was
> >>> mentioned as one of the commercial alternatives.
> >>>
> >>> I thought I'd reach out and see if I could provide you with any
> >>> information regarding our ExpeDat, high-performance data transfer,
> >>> software.  ExpeDat is a high-performance alternative to FTP and is
> >>> comprised of client and server software.  We do utilize UDP and then
> >>> have our own proprietary transport protocol (MTP/IP) piggy-back on top
> >>> which provides advanced flow-control, error-recovery, etc.
> >>>
> >>> We believe our claims to be realistic, unlike those pointed out in the
> >>> original posting citing some of our competitors.  To that end, we
> >>> typically see, on average across a high-speed WAN, roughly 7x faster
> >>> throughput and 10x greater reliability than traditional FTP.  Numbers
> >>> typically go up from there when you start talking about against SFTP,
> >>> scp, etc.
> >>>
> >>> If we can provide any additional insight for you and your colleagues,
> >>> please do not hesitate to contact me.
> >>>
> >>> Best,
> >>> Dana
> >>> -----------------------------------------------
> >>> J.P. Dana Merk
> >>> Director of Business Development
> >>> Data Expedition, Inc.
> >>> 617.500.0002, ext. 805 (office)
> >>> 203.668.9869 (mobile)
> >>> Skype: dana_merk
> >>> www.dataexpedition.com
> >>>
> >>>
> >>>
> >>
> >
> >