You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ftpserver-users@mina.apache.org by Charles Karow <ch...@karow.com> on 2009/10/12 18:15:09 UTC

IODataConnection.getDataInputStream()

I have written a  special-purpose, upload-only server by writing my own 
implementation of the STOR command, and I need access to the input 
stream of the data connection so I can process the incoming file. I 
prefer not to complete an implementation of my own file system -- for 
what I am doing, that would be much too complex and is not needed.

I have done this successfully with an early version of ftpserver by 
modifying the source code, but I would rather be able to use the 
published binaries as this would make it much easier to stay current 
with new releases.

IODataConnection.getDataInputStream() is currently private. I was 
wondering whether there would be any possibility of making it public. 
Would there be any adverse consequences?

Thanks!
Charles



Re: IODataConnection.getDataInputStream()

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, Oct 14, 2009 at 10:28 AM, David Latorre <dv...@gmail.com> wrote:
>  This is I what I first thought but I think this might imply  several
> risks in terms of performance, or the need to store the whole
> transferred file locally (be it in memory or disk). This is of course,
> if he cannot use PipedStreams.

As far as I can see he should be able to use PipedStreams, that was
what I was suggesting :-)

> -  If you are using different threads for the FTP transfer and the
> transfer to Amazon I guess you could use PipedStreams with our current
> code (I haven't looked at it actually).

Agreed

> If no one comes up with a solution for this, i don't think we should
> dismiss the possibility of exposing the input stream,  what do you
> think niklas?

I don't think we should do it as part of our public API. The public
API is DataConnection. Which would mean that any refactoring of
IODataConnection would potentially change how it works internally and
thus break this type of application.

/niklas

RE: implement a ftplet to limit the size of upload files

Posted by "Zhu, Yan-Bo (TSG-GDCC-CD)" <ya...@hp.com>.
It's really useful for me. thanks.
-- awaitUninterruptibly : it makes sure the 150 150_FILE_STATUS_OKAY command is sent before proceeding.
So, if 150 command is not sent before proceeding , is it a matter?

Thanks ,Bob Zhu 
---------------
>>
>>
I think your code is OK  except for  a  problem:
 - Restarting a failed transfer would mean that the user can safely
ignore your limit. To solve this, you could take into account the
skipLen value in order to prevent users from circumventing your
limits.


  The pieces of code you didn't understand:
  -- First block:              // 24-10-2007 - added check if PORT or
PASV is issued, see
//            // https://issues.apache.org/jira/browse/FTPSERVER-110

This block checks if a PORT or PASV command was issued previously. In
FTP, you must send a PORT or PASV command before any command that
sends data (stor, retr, mlsd etc.).

 -- awaitUninterruptibly : it makes sure the 150 150_FILE_STATUS_OKAY
command is sent before proceeding.
 --ServerFtpStatistics: this component keeps statistics of the
FTPServer usage. If you don't know it, you probably don't need it.
Still, I think your code doesn't break it in any way.



 2009/10/21 Zhu, Yan-Bo (TSG-GDCC-CD) <ya...@hp.com>:
> I'v figured out a implementation by copy the sourcecode in STOR.java to my Ftplet's onUpload method.
> The following the sourcecode ,pls notice the "//" at the start of some lines ,
> Some changes are needed:
> 1, LocalizedFtpReply.translate(....)method is not usable in ftplet,
> 2, create a new FilterOutputStream object and pass it to transferFromClient to limit the size .
> When I test from a ftp client ,it seems like that the result is what I want .
> But I'm not sure there is no bug because I'm not fully understand the code I commented by "//".
>
> Could someone give me some suggestions?
> Thanks.
> ::::::::::::::::::::::::::::
>                 try {
>
>            // get state variable
>            long skipLen = session.getFileOffset();
>
>            // argument check
>            String fileName = request.getArgument();
>            if (fileName == null) {
>                session
>                        .write(new DefaultFtpReply(
>                                        FtpReply.REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS,
>                                        "STOR REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS"));
>                return  FtpletResult.SKIP;
>            }
>
> //            // 24-10-2007 - added check if PORT or PASV is issued, see
> //            // https://issues.apache.org/jira/browse/FTPSERVER-110
> //            DataConnectionFactory connFactory = session.getDataConnection();
> //            if (connFactory instanceof IODataConnectionFactory) {
> //                InetAddress address = ((IODataConnectionFactory) connFactory)
> //                        .getInetAddress();
> //                if (address == null) {
> //                    session.write(new DefaultFtpReply(
> //                            FtpReply.REPLY_503_BAD_SEQUENCE_OF_COMMANDS,
> //                            "PORT or PASV must be issued first"));
> //                    return;
> //                }
> //            }
>
>            // get filename
>            FtpFile file = null;
>            try {
>                file = session.getFileSystemView().getFile(fileName);
>            } catch (Exception ex) {
>                System.out.println("Exception getting file object"+ ex);
>            }
>            if (file == null) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
>                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
>                return FtpletResult.SKIP;
>            }
>            fileName = file.getAbsolutePath();
>
>            // get permission
>            if (!file.isWritable()) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
>                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
>                return FtpletResult.SKIP;
>            }
>
>            // get data connection
>            session.write(
>                   new DefaultFtpReply(
>                            FtpReply.REPLY_150_FILE_STATUS_OKAY, "STOR.REPLY_150_FILE_STATUS_OKAY"))
> //               .awaitUninterruptibly(10000)
>                                ;
>
>            DataConnection dataConnection;
>            try {
>                dataConnection = session.getDataConnection().openConnection();
>            } catch (Exception e) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "STOR.REPLY_425_CANT_OPEN_DATA_CONNECTION"
>                        ));
>                return FtpletResult.SKIP;
>            }
>
>            // transfer data
>            boolean failure = false;
>            OutputStream outStream = null;
>            try {
>                outStream = file.createOutputStream(skipLen);
>                                final int writelimit = 2048;
>                long transSz = dataConnection.transferFromClient(
>                                        session, new FilterOutputStream( outStream){
>                                        private  int count =0;
>                                        public void write(int b) throws IOException {
>                                                        count++;
>                                                        if( count > writelimit ) throw new IOException ("Write too many bytes");
>                                                        out.write(b);
>                                        }
>
>                                });
>
>                // attempt to close the output stream so that errors in
>                // closing it will return an error to the client (FTPSERVER-119)
>                if(outStream != null) {
>                    outStream.close();
>                }
>
>                // notify the statistics component
> //                ServerFtpStatistics ftpStat = (ServerFtpStatistics) context
> //                        .getFtpStatistics();
> //                ftpStat.setUpload(session, file, transSz);
>
>
>            } catch (SocketException ex) {
>                failure = true;
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED,
>                        "STOR.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED"));
>            } catch (IOException ex) {
>                failure = true;
>                session
>                        .write(new DefaultFtpReply(
>                                        FtpReply.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN,
>                                        "STOR.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN"));
>            } finally {
>                // make sure we really close the output stream
>                outStream.close();
>            }
>
>            // if data transfer ok - send transfer complete message
>            if (!failure) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_226_CLOSING_DATA_CONNECTION, "STOR.REPLY_226_CLOSING_DATA_CONNECTION"));
>
>            }
>        } finally {
> //            session.resetState();
>            session.getDataConnection().closeDataConnection();
>        }
>
>                }catch(Exception e) {
>                        System.out.println(e);
>                }
>
>                return FtpletResult.SKIP;
>
>
>
> -------------------------------------------------------------------------------------------------
>>>Topic: implement a ftplet to limit the size of upload files
>
>>>Hi,
>
>>>I need to limit the size of upload files ,are there any good suggestions?
>>>Is implementing a ftplet possible to do this?
>
>>>Thanks for the help!
>>>Bob Zhu
>

Re: implement a ftplet to limit the size of upload files

Posted by David Latorre <dv...@gmail.com>.
I think your code is OK  except for  a  problem:
 - Restarting a failed transfer would mean that the user can safely
ignore your limit. To solve this, you could take into account the
skipLen value in order to prevent users from circumventing your
limits.


  The pieces of code you didn't understand:
  -- First block:              // 24-10-2007 - added check if PORT or
PASV is issued, see
//            // https://issues.apache.org/jira/browse/FTPSERVER-110

This block checks if a PORT or PASV command was issued previously. In
FTP, you must send a PORT or PASV command before any command that
sends data (stor, retr, mlsd etc.).

 -- awaitUninterruptibly : it makes sure the 150 150_FILE_STATUS_OKAY
command is sent before proceeding.
 --ServerFtpStatistics: this component keeps statistics of the
FTPServer usage. If you don't know it, you probably don't need it.
Still, I think your code doesn't break it in any way.



 2009/10/21 Zhu, Yan-Bo (TSG-GDCC-CD) <ya...@hp.com>:
> I'v figured out a implementation by copy the sourcecode in STOR.java to my Ftplet's onUpload method.
> The following the sourcecode ,pls notice the "//" at the start of some lines ,
> Some changes are needed:
> 1, LocalizedFtpReply.translate(....)method is not usable in ftplet,
> 2, create a new FilterOutputStream object and pass it to transferFromClient to limit the size .
> When I test from a ftp client ,it seems like that the result is what I want .
> But I'm not sure there is no bug because I'm not fully understand the code I commented by "//".
>
> Could someone give me some suggestions?
> Thanks.
> ::::::::::::::::::::::::::::
>                 try {
>
>            // get state variable
>            long skipLen = session.getFileOffset();
>
>            // argument check
>            String fileName = request.getArgument();
>            if (fileName == null) {
>                session
>                        .write(new DefaultFtpReply(
>                                        FtpReply.REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS,
>                                        "STOR REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS"));
>                return  FtpletResult.SKIP;
>            }
>
> //            // 24-10-2007 - added check if PORT or PASV is issued, see
> //            // https://issues.apache.org/jira/browse/FTPSERVER-110
> //            DataConnectionFactory connFactory = session.getDataConnection();
> //            if (connFactory instanceof IODataConnectionFactory) {
> //                InetAddress address = ((IODataConnectionFactory) connFactory)
> //                        .getInetAddress();
> //                if (address == null) {
> //                    session.write(new DefaultFtpReply(
> //                            FtpReply.REPLY_503_BAD_SEQUENCE_OF_COMMANDS,
> //                            "PORT or PASV must be issued first"));
> //                    return;
> //                }
> //            }
>
>            // get filename
>            FtpFile file = null;
>            try {
>                file = session.getFileSystemView().getFile(fileName);
>            } catch (Exception ex) {
>                System.out.println("Exception getting file object"+ ex);
>            }
>            if (file == null) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
>                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
>                return FtpletResult.SKIP;
>            }
>            fileName = file.getAbsolutePath();
>
>            // get permission
>            if (!file.isWritable()) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
>                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
>                return FtpletResult.SKIP;
>            }
>
>            // get data connection
>            session.write(
>                   new DefaultFtpReply(
>                            FtpReply.REPLY_150_FILE_STATUS_OKAY, "STOR.REPLY_150_FILE_STATUS_OKAY"))
> //               .awaitUninterruptibly(10000)
>                                ;
>
>            DataConnection dataConnection;
>            try {
>                dataConnection = session.getDataConnection().openConnection();
>            } catch (Exception e) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "STOR.REPLY_425_CANT_OPEN_DATA_CONNECTION"
>                        ));
>                return FtpletResult.SKIP;
>            }
>
>            // transfer data
>            boolean failure = false;
>            OutputStream outStream = null;
>            try {
>                outStream = file.createOutputStream(skipLen);
>                                final int writelimit = 2048;
>                long transSz = dataConnection.transferFromClient(
>                                        session, new FilterOutputStream( outStream){
>                                        private  int count =0;
>                                        public void write(int b) throws IOException {
>                                                        count++;
>                                                        if( count > writelimit ) throw new IOException ("Write too many bytes");
>                                                        out.write(b);
>                                        }
>
>                                });
>
>                // attempt to close the output stream so that errors in
>                // closing it will return an error to the client (FTPSERVER-119)
>                if(outStream != null) {
>                    outStream.close();
>                }
>
>                // notify the statistics component
> //                ServerFtpStatistics ftpStat = (ServerFtpStatistics) context
> //                        .getFtpStatistics();
> //                ftpStat.setUpload(session, file, transSz);
>
>
>            } catch (SocketException ex) {
>                failure = true;
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED,
>                        "STOR.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED"));
>            } catch (IOException ex) {
>                failure = true;
>                session
>                        .write(new DefaultFtpReply(
>                                        FtpReply.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN,
>                                        "STOR.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN"));
>            } finally {
>                // make sure we really close the output stream
>                outStream.close();
>            }
>
>            // if data transfer ok - send transfer complete message
>            if (!failure) {
>                session.write(new DefaultFtpReply(
>                        FtpReply.REPLY_226_CLOSING_DATA_CONNECTION, "STOR.REPLY_226_CLOSING_DATA_CONNECTION"));
>
>            }
>        } finally {
> //            session.resetState();
>            session.getDataConnection().closeDataConnection();
>        }
>
>                }catch(Exception e) {
>                        System.out.println(e);
>                }
>
>                return FtpletResult.SKIP;
>
>
>
> -------------------------------------------------------------------------------------------------
>>>Topic: implement a ftplet to limit the size of upload files
>
>>>Hi,
>
>>>I need to limit the size of upload files ,are there any good suggestions?
>>>Is implementing a ftplet possible to do this?
>
>>>Thanks for the help!
>>>Bob Zhu
>

Re: implement a ftplet to limit the size of upload files

Posted by "Zhu, Yan-Bo (TSG-GDCC-CD)" <ya...@hp.com>.
I'v figured out a implementation by copy the sourcecode in STOR.java to my Ftplet's onUpload method.
The following the sourcecode ,pls notice the "//" at the start of some lines ,
Some changes are needed:
1, LocalizedFtpReply.translate(....)method is not usable in ftplet, 
2, create a new FilterOutputStream object and pass it to transferFromClient to limit the size .
When I test from a ftp client ,it seems like that the result is what I want .
But I'm not sure there is no bug because I'm not fully understand the code I commented by "//".

Could someone give me some suggestions?
Thanks.
::::::::::::::::::::::::::::
		 try {

            // get state variable
            long skipLen = session.getFileOffset();

            // argument check
            String fileName = request.getArgument();
            if (fileName == null) {
                session
                        .write(new DefaultFtpReply(
                                        FtpReply.REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS,
                                        "STOR REPLY_501_SYNTAX_ERROR_IN_PARAMETERS_OR_ARGUMENTS"));
                return  FtpletResult.SKIP;
            }
	
//            // 24-10-2007 - added check if PORT or PASV is issued, see
//            // https://issues.apache.org/jira/browse/FTPSERVER-110
//            DataConnectionFactory connFactory = session.getDataConnection();
//            if (connFactory instanceof IODataConnectionFactory) {
//                InetAddress address = ((IODataConnectionFactory) connFactory)
//                        .getInetAddress();
//                if (address == null) {
//                    session.write(new DefaultFtpReply(
//                            FtpReply.REPLY_503_BAD_SEQUENCE_OF_COMMANDS,
//                            "PORT or PASV must be issued first"));
//                    return;
//                }
//            }
        
            // get filename
            FtpFile file = null;
            try {
                file = session.getFileSystemView().getFile(fileName);
            } catch (Exception ex) {
                System.out.println("Exception getting file object"+ ex);
            }
            if (file == null) {
                session.write(new DefaultFtpReply(
                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
                return FtpletResult.SKIP;
            }
            fileName = file.getAbsolutePath();

            // get permission
            if (!file.isWritable()) {
                session.write(new DefaultFtpReply(
                        FtpReply.REPLY_550_REQUESTED_ACTION_NOT_TAKEN,
                        "STOR.REPLY_550_REQUESTED_ACTION_NOT_TAKEN"));
                return FtpletResult.SKIP;
            }

            // get data connection
            session.write(
                   new DefaultFtpReply(
                            FtpReply.REPLY_150_FILE_STATUS_OKAY, "STOR.REPLY_150_FILE_STATUS_OKAY"))
//               .awaitUninterruptibly(10000)
				;

            DataConnection dataConnection;
            try {
                dataConnection = session.getDataConnection().openConnection();
            } catch (Exception e) {
                session.write(new DefaultFtpReply(
                        FtpReply.REPLY_425_CANT_OPEN_DATA_CONNECTION, "STOR.REPLY_425_CANT_OPEN_DATA_CONNECTION"
                        ));
                return FtpletResult.SKIP;
            }

            // transfer data
            boolean failure = false;
            OutputStream outStream = null;
            try {
                outStream = file.createOutputStream(skipLen);
				final int writelimit = 2048;
                long transSz = dataConnection.transferFromClient(
					session, new FilterOutputStream( outStream){
					private  int count =0;
					public void write(int b) throws IOException {
							count++;
							if( count > writelimit ) throw new IOException ("Write too many bytes");
							out.write(b);
					}
					 
				});

                // attempt to close the output stream so that errors in 
                // closing it will return an error to the client (FTPSERVER-119) 
                if(outStream != null) {
                    outStream.close();
                }

                // notify the statistics component
//                ServerFtpStatistics ftpStat = (ServerFtpStatistics) context
//                        .getFtpStatistics();
//                ftpStat.setUpload(session, file, transSz);
		
                
            } catch (SocketException ex) {
                failure = true;
                session.write(new DefaultFtpReply(
                        FtpReply.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED,
                        "STOR.REPLY_426_CONNECTION_CLOSED_TRANSFER_ABORTED"));
            } catch (IOException ex) {
                failure = true;
                session
                        .write(new DefaultFtpReply(
                                        FtpReply.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN,
                                        "STOR.REPLY_551_REQUESTED_ACTION_ABORTED_PAGE_TYPE_UNKNOWN"));
            } finally {
                // make sure we really close the output stream
                outStream.close();
            }

            // if data transfer ok - send transfer complete message
            if (!failure) {
                session.write(new DefaultFtpReply(
                        FtpReply.REPLY_226_CLOSING_DATA_CONNECTION, "STOR.REPLY_226_CLOSING_DATA_CONNECTION"));

            }
        } finally {
//            session.resetState();
            session.getDataConnection().closeDataConnection();
        }

		}catch(Exception e) {
			System.out.println(e);
		}
		
		return FtpletResult.SKIP;



-------------------------------------------------------------------------------------------------
>>Topic: implement a ftplet to limit the size of upload files

>>Hi,

>>I need to limit the size of upload files ,are there any good suggestions?
>>Is implementing a ftplet possible to do this?

>>Thanks for the help!
>>Bob Zhu

implement a ftplet to limit the size of upload files

Posted by "Zhu, Yan-Bo (TSG-GDCC-CD)" <ya...@hp.com>.
Hi,

I need to limit the size of upload files ,are there any good suggestions?
Is implementing a ftplet possible to do this?

Thanks for the help!
Bob Zhu

Re: IODataConnection.getDataInputStream()

Posted by Charles Karow <ch...@karow.com>.
Thanks everyone for the discussion and suggestions!

I did some digging and learned how to use PipedStreams and this seems to 
be working well for me.

I, too, am concerned about the reliability problems with S3, but 
experience over the past year and a half is that the failure rate is 
less than 1% and has been improving. My users seem to be satisfied with 
an occasional error. If I get an S3 failure, I return an error to the 
client. My concern about buffering whole files is the performance and 
memory requirements that would limit the number of users that could be 
supported.

Thanks for the help!
Charles


Brad McEvoy wrote:

>hey,
>
>just one experience i've had with s3 which might impact on the design.
>i've found that uploads into s3 (when under load) fail surprisingly
>frequently, at a rate of about 1 in a hundred. the s3 docs say that
>upload failures are a normal and an expected part of the service, but i
>didnt really think it would fail that often.
>
>so my conclusion is that i have to buffer uploads and then load into s3
>asynchronously to ensure that end users don't get impacted. if you do
>this then you end up doing two seperate file transfers anyway, so
>there's no big deal about going inputstream to inputstream.
>
>another consideration is that uploads to s3 are non-transactional. so if
>you stream directly from end user to s3, and the end user aborts the
>upload, then you'll end up with a completely useless part of a file
>stored in s3. if you buffer locally and only load into s3 on success you
>won't have this problem.
>
>cheers,
>brad
>
>On Wed, 14 Oct 2009 10:28 +0200, "David Latorre" <dv...@gmail.com>
>wrote:
>  
>
>>2009/10/13 Niklas Gustavsson <ni...@protocol7.com>:
>>    
>>
>>>On Tue, Oct 13, 2009 at 6:21 AM, Charles Karow <ch...@karow.com> wrote:
>>>      
>>>
>>>>I am using ftpserver to provide a standard way for people to upload files to
>>>>a "bucket" on Amazon's S3 service. My users will always be uploading files
>>>>in binary mode. I am using code from Amazon that takes an InputStream and
>>>>uses it to stream the data to Amazon's servers. Amazon's code does not
>>>>expose an OutputStream.
>>>>
>>>>transferFromClient takes an OutputStream and I do not have access to an
>>>>OutputStream.
>>>>        
>>>>
>>>Sounds like this could be solved by an adapter stream which gets
>>>written to by DataConnection and is read by S3. Or I might be missing
>>>something?
>>>      
>>>
>> This is I what I first thought but I think this might imply  several
>>risks in terms of performance, or the need to store the whole
>>transferred file locally (be it in memory or disk). This is of course,
>>if he cannot use PipedStreams.
>>
>>I may not be thinking correctly now but for a solution:
>>
>>-  If you are using different threads for the FTP transfer and the
>>transfer to Amazon I guess you could use PipedStreams with our current
>>code (I haven't looked at it actually).
>>
>> - Otherwise, maybe someone on this list can tell us what their
>>approach is. I think some of them are using S3.
>>
>>If no one comes up with a solution for this, i don't think we should
>>dismiss the possibility of exposing the input stream,  what do you
>>think niklas?
>>    
>>


Re: IODataConnection.getDataInputStream()

Posted by Brad McEvoy <br...@bradmcevoy.com>.
hey,

just one experience i've had with s3 which might impact on the design.
i've found that uploads into s3 (when under load) fail surprisingly
frequently, at a rate of about 1 in a hundred. the s3 docs say that
upload failures are a normal and an expected part of the service, but i
didnt really think it would fail that often.

so my conclusion is that i have to buffer uploads and then load into s3
asynchronously to ensure that end users don't get impacted. if you do
this then you end up doing two seperate file transfers anyway, so
there's no big deal about going inputstream to inputstream.

another consideration is that uploads to s3 are non-transactional. so if
you stream directly from end user to s3, and the end user aborts the
upload, then you'll end up with a completely useless part of a file
stored in s3. if you buffer locally and only load into s3 on success you
won't have this problem.

cheers,
brad

On Wed, 14 Oct 2009 10:28 +0200, "David Latorre" <dv...@gmail.com>
wrote:
> 2009/10/13 Niklas Gustavsson <ni...@protocol7.com>:
> > On Tue, Oct 13, 2009 at 6:21 AM, Charles Karow <ch...@karow.com> wrote:
> >> I am using ftpserver to provide a standard way for people to upload files to
> >> a "bucket" on Amazon's S3 service. My users will always be uploading files
> >> in binary mode. I am using code from Amazon that takes an InputStream and
> >> uses it to stream the data to Amazon's servers. Amazon's code does not
> >> expose an OutputStream.
> >>
> >> transferFromClient takes an OutputStream and I do not have access to an
> >> OutputStream.
> >
> > Sounds like this could be solved by an adapter stream which gets
> > written to by DataConnection and is read by S3. Or I might be missing
> > something?
> 
>  This is I what I first thought but I think this might imply  several
> risks in terms of performance, or the need to store the whole
> transferred file locally (be it in memory or disk). This is of course,
> if he cannot use PipedStreams.
> 
> I may not be thinking correctly now but for a solution:
> 
> -  If you are using different threads for the FTP transfer and the
> transfer to Amazon I guess you could use PipedStreams with our current
> code (I haven't looked at it actually).
> 
>  - Otherwise, maybe someone on this list can tell us what their
> approach is. I think some of them are using S3.
> 
> If no one comes up with a solution for this, i don't think we should
> dismiss the possibility of exposing the input stream,  what do you
> think niklas?
-- 
  Brad McEvoy
  brad@bradmcevoy.com


Re: IODataConnection.getDataInputStream()

Posted by David Latorre <dv...@gmail.com>.
2009/10/13 Niklas Gustavsson <ni...@protocol7.com>:
> On Tue, Oct 13, 2009 at 6:21 AM, Charles Karow <ch...@karow.com> wrote:
>> I am using ftpserver to provide a standard way for people to upload files to
>> a "bucket" on Amazon's S3 service. My users will always be uploading files
>> in binary mode. I am using code from Amazon that takes an InputStream and
>> uses it to stream the data to Amazon's servers. Amazon's code does not
>> expose an OutputStream.
>>
>> transferFromClient takes an OutputStream and I do not have access to an
>> OutputStream.
>
> Sounds like this could be solved by an adapter stream which gets
> written to by DataConnection and is read by S3. Or I might be missing
> something?

 This is I what I first thought but I think this might imply  several
risks in terms of performance, or the need to store the whole
transferred file locally (be it in memory or disk). This is of course,
if he cannot use PipedStreams.

I may not be thinking correctly now but for a solution:

-  If you are using different threads for the FTP transfer and the
transfer to Amazon I guess you could use PipedStreams with our current
code (I haven't looked at it actually).

 - Otherwise, maybe someone on this list can tell us what their
approach is. I think some of them are using S3.

If no one comes up with a solution for this, i don't think we should
dismiss the possibility of exposing the input stream,  what do you
think niklas?

Re: IODataConnection.getDataInputStream()

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Tue, Oct 13, 2009 at 6:21 AM, Charles Karow <ch...@karow.com> wrote:
> I am using ftpserver to provide a standard way for people to upload files to
> a "bucket" on Amazon's S3 service. My users will always be uploading files
> in binary mode. I am using code from Amazon that takes an InputStream and
> uses it to stream the data to Amazon's servers. Amazon's code does not
> expose an OutputStream.
>
> transferFromClient takes an OutputStream and I do not have access to an
> OutputStream.

Sounds like this could be solved by an adapter stream which gets
written to by DataConnection and is read by S3. Or I might be missing
something?

/niklas

Re: IODataConnection.getDataInputStream()

Posted by Charles Karow <ch...@karow.com>.
I am using ftpserver to provide a standard way for people to upload 
files to a "bucket" on Amazon's S3 service. My users will always be 
uploading files in binary mode. I am using code from Amazon that takes 
an InputStream and uses it to stream the data to Amazon's servers. 
Amazon's code does not expose an OutputStream.

transferFromClient takes an OutputStream and I do not have access to an 
OutputStream.

-Charles


Niklas Gustavsson wrote:

>On Mon, Oct 12, 2009 at 6:15 PM, Charles Karow <ch...@karow.com> wrote:
>  
>
>>I have written a  special-purpose, upload-only server by writing my own
>>implementation of the STOR command, and I need access to the input stream of
>>the data connection so I can process the incoming file. I prefer not to
>>complete an implementation of my own file system -- for what I am doing,
>>that would be much too complex and is not needed.
>>    
>>
>
>Could you go into detail why the transferFromClient method is not
>applicable for your use case?
>
>/niklas
>
>  
>

Re: IODataConnection.getDataInputStream()

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Mon, Oct 12, 2009 at 6:15 PM, Charles Karow <ch...@karow.com> wrote:
> I have written a  special-purpose, upload-only server by writing my own
> implementation of the STOR command, and I need access to the input stream of
> the data connection so I can process the incoming file. I prefer not to
> complete an implementation of my own file system -- for what I am doing,
> that would be much too complex and is not needed.

Could you go into detail why the transferFromClient method is not
applicable for your use case?

/niklas