You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Geoffroy EMPAIN <em...@hotmail.com> on 2013/05/05 09:47:17 UTC
FTPClient, resume an interrupted upload?
Hello everyone,
I'm currently working on an App that uploads large PDFs to a server. The app works fine, but everytime, since the PDFs are so large (25MB), the upload takes a while, and usually after let's say 30 or 40 minutes, I get a "socketException : broken pipe". I believe that this is due to a timeout or disconnect from the server (the server cuts the connection I guess), so I moved the upload routine in a loop that has a try / catch. When an exception is thrown, I try to reconnect. It works fine. The upload starts where it stopped and completes.
Problem? Well, since the upload is "broken" in two parts (or more if any other connection loss occurs), the file uploaded is broken too! I mean that's definatley normal and I unerstand why that happens, but what I'd like to know is how you can keep the upload on "hold" while trying to reconnect. I just want to be able to do a complete upload even if there are reconnections. I want my PDF to uploaded completely. Here is the code I have :
// In
inputStream = new FileInputStream(localFile);
// For upload loop
byte[] bytesIn = new byte[4096];
int read = 0;
// Loop until job is not completed (will try to reconnect if any exception is thrown)
boolean completed = false;
while (!completed){
try{
// Out
OutputStream outputStream = ftpClient.storeFileStream(remoteFile);
// Transfer
while ((read = inputStream.read(bytesIn)) != -1) {
outputStream.write(bytesIn, 0, read);
transfered += read;
}
// Closing streams
inputStream.close();
outputStream.close();
// Final information
completed = ftpClient.completePendingCommand();
if (completed) System.out.println("Done");
else System.out.println("Failure.");
completed = true;
} // end try
catch (Exception e) {
// Telling the user
System.out.println("Trying to reconnect...");
Thread.sleep(1000);
// Try to reconnect
ftpClient.connect(server, port);
success = ftpClient.login(user, pass);
if (success) {
System.out.println("Connection : OK");
Thread.sleep(1000);
} else {
System.out.println("Failed to connect");
}
// Set passive mode and file type to binary
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
} // end catch
} // end loop
Any help will be greatly apreciated! Thanks to the community!
Best regards;
Geoffrey
Re: FTPClient, resume an interrupted upload?
Posted by sebb <se...@gmail.com>.
I think you need to discover why the error is occuring and fix that.
I doubt it will be possible to reuse the existing connection once an
error occurs.
However if that proves impossible the FTP protocol does support the
REST command which tells the server where to start uploading.
Obviously you have to skip the same portion of the file when you then
do the transfer.
On 8 May 2013 17:58, G E <em...@hotmail.com> wrote:
> Alright, so I tried the last algorithme I sent you, and now I get this ^^ :
>
> 68 %
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
> at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at java.io.BufferedReader.fill(BufferedReader.java:136)
> at java.io.BufferedReader.read(BufferedReader.java:157)
> at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:483)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:556)
> at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:605)
> at org.apache.commons.net.ftp.FTP.pasv(FTP.java:956)
> at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:806)
> at org.apache.commons.net.ftp.FTPClient._storeFileStream(FTPClient.java:657)
> at org.apache.commons.net.ftp.FTPClient.__storeFileStream(FTPClient.java:648)
> at org.apache.commons.net.ftp.FTPClient.appendFileStream(FTPClient.java:1995)
> at Main.startUpload(Main.java:356)
> at Main.uploadFiles(Main.java:240)
> at Main$2.actionPerformed(Main.java:154)
> at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1882)
> at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2202)
> at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
> at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
> at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:246)
> at java.awt.Component.processMouseEvent(Component.java:5617)
> at javax.swing.JComponent.processMouseEvent(JComponent.java:3129)
> at java.awt.Component.processEvent(Component.java:5382)
> at java.awt.Container.processEvent(Container.java:2010)
> at java.awt.Component.dispatchEventImpl(Component.java:4083)
> at java.awt.Container.dispatchEventImpl(Container.java:2068)
> at java.awt.Component.dispatchEvent(Component.java:3918)
> at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4256)
> at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3936)
> at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3866)
> at java.awt.Container.dispatchEventImpl(Container.java:2054)
> at java.awt.Window.dispatchEventImpl(Window.java:1801)
> at java.awt.Component.dispatchEvent(Component.java:3918)
> at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:501)
> at java.awt.EventQueue.access$000(EventQueue.java:80)
> at java.awt.EventQueue$1.run(EventQueue.java:462)
> at java.awt.EventQueue$1.run(EventQueue.java:461)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
> at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:95)
> at java.awt.EventQueue$2.run(EventQueue.java:476)
> at java.awt.EventQueue$2.run(EventQueue.java:475)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
> at java.awt.EventQueue.dispatchEvent(EventQueue.java:473)
> at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
> at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
> at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
> at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
> at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
>
> So it sounds like it's not the write methode that thrws the exception :/
>
> Any idea?
>
> Best regards, G. E.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: FTPClient, resume an interrupted upload?
Posted by G E <em...@hotmail.com>.
Alright, so I tried the last algorithme I sent you, and now I get this ^^ :
68 %
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:483)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:556)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:605)
at org.apache.commons.net.ftp.FTP.pasv(FTP.java:956)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:806)
at org.apache.commons.net.ftp.FTPClient._storeFileStream(FTPClient.java:657)
at org.apache.commons.net.ftp.FTPClient.__storeFileStream(FTPClient.java:648)
at org.apache.commons.net.ftp.FTPClient.appendFileStream(FTPClient.java:1995)
at Main.startUpload(Main.java:356)
at Main.uploadFiles(Main.java:240)
at Main$2.actionPerformed(Main.java:154)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1882)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2202)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:246)
at java.awt.Component.processMouseEvent(Component.java:5617)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3129)
at java.awt.Component.processEvent(Component.java:5382)
at java.awt.Container.processEvent(Container.java:2010)
at java.awt.Component.dispatchEventImpl(Component.java:4083)
at java.awt.Container.dispatchEventImpl(Container.java:2068)
at java.awt.Component.dispatchEvent(Component.java:3918)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4256)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3936)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3866)
at java.awt.Container.dispatchEventImpl(Container.java:2054)
at java.awt.Window.dispatchEventImpl(Window.java:1801)
at java.awt.Component.dispatchEvent(Component.java:3918)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:501)
at java.awt.EventQueue.access$000(EventQueue.java:80)
at java.awt.EventQueue$1.run(EventQueue.java:462)
at java.awt.EventQueue$1.run(EventQueue.java:461)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:95)
at java.awt.EventQueue$2.run(EventQueue.java:476)
at java.awt.EventQueue$2.run(EventQueue.java:475)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:473)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
So it sounds like it's not the write methode that thrws the exception :/
Any idea?
Best regards, G. E.
RE: FTPClient, resume an interrupted upload?
Posted by G E <em...@hotmail.com>.
Hi and thank you for the reply. Is this what you mean : (?)
OutputStream outputStream = ftpClient.storeFileStream(remoteFile);
while ((read = inputStream.read(bytesIn)) != -1) {
try {
outputStream.write(bytesIn, 0, read);
} catch (Exception e) {
OutputStream os = ftpClient.appendFileStream(remoteFile);
os.write(bytesIn, 0, read);
os.close;
}
}
Best regards.
> From: mcucchiara@apache.org
> Date: Tue, 7 May 2013 12:34:07 +0200
> Subject: Re: FTPClient, resume an interrupted upload?
> To: user@commons.apache.org
>
> Since the exception is probably thrown on outputStream.write method, you should:
> * create a new outputStream using appendFileStream method [1].
> * try to append the previously read bytesIn value
>
> > OutputStream outputStream = ftpClient.storeFileStream(remoteFile);
> >
> > // Transfer
> > while ((read = inputStream.read(bytesIn)) != -1) {
> > outputStream.write(bytesIn, 0, read);
> > transfered += read;
> > }
>
> [1] http://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#appendFileStream(java.lang.String)
> Twitter :http://www.twitter.com/m_cucchiara
> G+ :https://plus.google.com/107903711540963855921
> Linkedin :http://www.linkedin.com/in/mauriziocucchiara
> VisualizeMe: http://vizualize.me/maurizio.cucchiara?r=maurizio.cucchiara
>
> Maurizio Cucchiara
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Re: FTPClient, resume an interrupted upload?
Posted by Maurizio Cucchiara <mc...@apache.org>.
Since the exception is probably thrown on outputStream.write method, you should:
* create a new outputStream using appendFileStream method [1].
* try to append the previously read bytesIn value
> OutputStream outputStream = ftpClient.storeFileStream(remoteFile);
>
> // Transfer
> while ((read = inputStream.read(bytesIn)) != -1) {
> outputStream.write(bytesIn, 0, read);
> transfered += read;
> }
[1] http://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#appendFileStream(java.lang.String)
Twitter :http://www.twitter.com/m_cucchiara
G+ :https://plus.google.com/107903711540963855921
Linkedin :http://www.linkedin.com/in/mauriziocucchiara
VisualizeMe: http://vizualize.me/maurizio.cucchiara?r=maurizio.cucchiara
Maurizio Cucchiara
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: FTPClient, resume an interrupted upload?
Posted by G E <em...@hotmail.com>.
Hello there,
thanks a lot for the reply. I set the keep alive timeout with "setControlKeepAliveTimeout(300)" but it did not work, here is the exception thrown :
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.commons.net.io.SocketOutputStream.write(SocketOutputStream.java:71)
at Main.startUpload(Main.java:349)
at Main.uploadFiles(Main.java:239)
at Main$2.actionPerformed(Main.java:153)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1882)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2202)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:420)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:258)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:246)
at java.awt.Component.processMouseEvent(Component.java:5617)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3129)
at java.awt.Component.processEvent(Component.java:5382)
at java.awt.Container.processEvent(Container.java:2010)
at java.awt.Component.dispatchEventImpl(Component.java:4083)
at java.awt.Container.dispatchEventImpl(Container.java:2068)
at java.awt.Component.dispatchEvent(Component.java:3918)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4256)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:3936)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:3866)
at java.awt.Container.dispatchEventImpl(Container.java:2054)
at java.awt.Window.dispatchEventImpl(Window.java:1801)
at java.awt.Component.dispatchEvent(Component.java:3918)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:501)
at java.awt.EventQueue.access$000(EventQueue.java:80)
at java.awt.EventQueue$1.run(EventQueue.java:462)
at java.awt.EventQueue$1.run(EventQueue.java:461)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:95)
at java.awt.EventQueue$2.run(EventQueue.java:476)
at java.awt.EventQueue$2.run(EventQueue.java:475)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:84)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:473)
at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
The upload always fails when I ge to either 65% or 69% of the upload.
Thanks for the support!
Best regards,
G.E.
> Date: Sun, 5 May 2013 18:10:05 +0200
> Subject: Re: FTPClient, resume an interrupted upload?
> From: mcucchiara@apache.org
> To: user@commons.apache.org
>
> By incrementing the keep alive timeout you should fix the issue.
> Anyway the code you posted could not work:
> * the output stream doesn't append to the file (there is a method for it)
> * the exception is probably thrown on outputStream.write, so you should try
> to write again the sa me byte array
Re: FTPClient, resume an interrupted upload?
Posted by Maurizio Cucchiara <mc...@apache.org>.
By incrementing the keep alive timeout you should fix the issue.
Anyway the code you posted could not work:
* the output stream doesn't append to the file (there is a method for it)
* the exception is probably thrown on outputStream.write, so you should try
to write again the sa me byte array