You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by "Seganti, Michael (DOF)" <Se...@finance.nyc.gov> on 2016/11/17 17:19:08 UTC
[net] FTPClient Transfer succeeds but fails in cleanUp
I'm having a sporadic issue transferring files to a Mainframe, was wondering if anyone else has run into this. I'm trying to figure out if this is a Mainframe issue, or a net commons issue.
I'm using Java 1.7 and commons-net-3.3.jar
The file transfer completes, but it looks like either the 200 OK is sent over the wrong connection, or is not properly being parsed and is causing the cleanup to fail.
According to the reply: IBM FTP CS V2R2 is the version where the issue occurs.
I have a second mainframe that is running IBM FTP CS V1R13 that doesn't seem to have this issue
In a normal transfer, you would see:
250 Transfer completed successfully.
File Transfer Completed Successfully at: 2016-11-17 04:30:51.569
QUIT
221 Quit command received. Goodbye.
This is what we are getting on the failures:
250 Transfer completed successfully.
200 OK
File Transfer Failed at: 2016-11-17 04:36:50.246
Error Occurred Transmitting File to Remote System, aborting...
Read timed out
Here's the full log from the failed transfer:
Using FTPS Connection with Protocol: TLS and Explicit Security Request
220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
220 Connection will close if idle for more than 5 minutes.
AUTH TLS
234 Security environment established - ready for negotiation
Connected to mvsp.xxx on xxxx
PBSZ 0
200 Protection buffer size accepted
PROT P
200 Data connection protection set to private
USER *******
331 Send password please.
PASS *******
230 C836PTS is logged on. Working directory is "C836PTS.".
PWD
257 "'C836PTS.'" is working directory.
Using a Packet Size of: 4096
Control KeepAliveTimeOut: 60
Control KeepAliveReplyTimeOut: 5000
FEAT
211- Extensions supported
AUTH TLS
PBSZ
PROT
211 End
...FTPClientUtil Connected Successfully!
SITE TRAIL
200 SITE command was accepted
SITE CYLINDERS PRIMARY=2100 SECONDARY=250
200 SITE command was accepted
SITE RECFM=FB LRECL=1915 BLKSIZE=30640
200 SITE command was accepted
SITE CONDDISP=delete
200 SITE command was accepted
Sending:
Local: d:\LargeExtracts\full.txt
Remote: 'A836.ORG.CAMA2.FULL'
Starting at: 2016-11-17 09:27:34.798
Total Bytes To Send: 2101359588
PASV
227 Entering Passive Mode (xx,xx,x,xx,xx,xx)
STOR 'A836.ORG.CAMA2.FULL'
125 Storing data set A836.ORG.CAMA2.FULL
2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred: 105070592
2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred: 210137088
...
2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred: 2101359588
250 Transfer completed successfully.
200 OK
File Transfer Failed at: 2016-11-17 09:32:26.127
Error Occurred Transmitting File to Remote System, aborting...
Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:304)
at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3697)
at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:669)
at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:624)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:1976)
at gov.nyc.ftputils.FTPConnection.sendFile(FTPConnection.java:695)
at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:54)
at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:96)
at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(FTPPropertyProcessor.java:126)
at gov.nyc.ftputils.FTPClientUtil.main(FTPClientUtil.java:378)
FTPClientUtil has Failed
QUIT
221 Quit command received. Goodbye.
Here's a code snippet of the transfer code:
try{
if (trace){
System.out.println("Total Bytes To Send: " + localFile.length());
CopyStreamListener listener = new ProgressStreamListener(localFile.length());
ftp.setCopyStreamListener(listener);
}
InputStream input = new FileInputStream(localFileName);
boolean result = ftp.storeFile(tempRemoteFileName, input);
input.close();
if (result){
System.out.println("\tFile Transfer Completed Successfully at: " + sdf.format(Calendar.getInstance().getTime()));
...
}
else
System.out.println("\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
return result;
}catch (CopyStreamException cse){
System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
System.err.println("Error Occurred Transmitting File to Remote System, aborting...\n");
cse.printStackTrace(System.err);
System.err.println("\n\nIOException Stack Trace that Caused the Error:\n");
cse.getIOException().printStackTrace(System.err);
return false;
}catch (IOException ioe){
System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
System.err.println("Error Occurred Transmitting File to Remote System, aborting...\n");
System.err.println(ioe.getMessage());
ioe.printStackTrace(System.err);
return false;
}catch (Exception e){
System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime()));
System.err.println("Error Occurred Transmitting File to Remote System, aborting...\n");
e.printStackTrace(System.err);
return false;
}
}
Thanks,
Mike...
RE: [net] FTPClient Transfer succeeds but fails in cleanUp
Posted by "Seganti, Michael (DOF)" <Se...@finance.nyc.gov>.
Same code on two different source hosts going to the same destination fails on occasion. I've never had an issue with any other destination hosts using this code (5 years?).
I had to add a bit to get the example code to work (pbsz and prot, as well as site commands), but I was able to successfully get the file transferred.
A little background on my issue, the mainframe I'm sending to is fairly saturated, and I suspected the issue was caused by that.
I started playing around with the control keepalive timeout's, which my understanding is, that a NOOP gets sent over the control connection every keepalive timeout period.
In my case, I override it by default to 4:30. I thought maybe the control connection was closing (timed out), and I wasn't receiving the successful reply, so I dropped it to 60 seconds.
I have one 2GB file which I can get to fail consistently using my code. When I set the keepAliveTimout (-k 60) using the FTPClientExample, it also fails in the same way.
It's odd that the protocol listener doesn't print the result, even though the file transmission completed.
Command:
java -cp commons-net-3.5.jar;. FTPClientExample -p false -# -s -k 60 mvsp.xxx:2121 C836PTS password 'A836.ORG.CAMA2.FULL' d:\LargeExtracts\full.txt
FTPClientExample Log Snippet:
#####################java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
at FTPClientExample.main(FTPClientExample.java:415)
-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: Thursday, November 17, 2016 2:49 PM
To: Commons Users List
Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
Does the same code fail when sending the same file to a different host?
Have you tried using the example utility?
http://commons.apache.org/proper/commons-net/#Examples
This exercises most of the FTP methods.
On 17 November 2016 at 18:01, Seganti, Michael (DOF) <Se...@finance.nyc.gov> wrote:
> I just updated to 3.5, and this file is consistently failing.
>
> I no longer receive the
>
> 250 Transfer completed successfully.
> 200 OK
>
> Replies.
>
> Log Snippet:
>
> 2016-11-17 12:56:33.870 - 100 Percent Complete, Total Bytes
> Transferred: 2101359588
>
> File Transfer Failed at: 2016-11-17 12:56:38.885 Error
> Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at sun.security.ssl.InputRecord.readFully(Unknown Source)
> at sun.security.ssl.InputRecord.read(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> at sun.security.ssl.AppInputStream.read(Unknown Source)
> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.fill(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
> at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
> at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
> at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
> at gov.nyc.ftputils.FTPConnection.sendFile(FTPConnection.java:695)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:54)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:96)
> at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(FTPPropertyProcessor.java:126)
> at gov.nyc.ftputils.FTPClientUtil.main(FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> -----Original Message-----
> From: Seganti, Michael (DOF) [mailto:SegantiM@finance.nyc.gov]
> Sent: Thursday, November 17, 2016 12:34 PM
> To: Commons Users List
> Subject: RE: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> I need to do a quick test prior to pushing to production, but yes, that's my next step.
>
> I didn't see anything that stood out in the release notes. It would make life easier if this was reproduceable...
>
> -----Original Message-----
> From: Gary Gregory [mailto:garydgregory@gmail.com]
> Sent: Thursday, November 17, 2016 12:31 PM
> To: Commons Users List
> Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> Can you try version 3.5?
>
> Gary
>
> On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) < SegantiM@finance.nyc.gov> wrote:
>
>> I'm having a sporadic issue transferring files to a Mainframe, was
>> wondering if anyone else has run into this. I'm trying to figure out
>> if this is a Mainframe issue, or a net commons issue.
>>
>> I'm using Java 1.7 and commons-net-3.3.jar
>>
>> The file transfer completes, but it looks like either the 200 OK is
>> sent over the wrong connection, or is not properly being parsed and
>> is causing the cleanup to fail.
>>
>> According to the reply: IBM FTP CS V2R2 is the version where the
>> issue occurs.
>>
>> I have a second mainframe that is running IBM FTP CS V1R13 that
>> doesn't seem to have this issue
>>
>> In a normal transfer, you would see:
>>
>> 250 Transfer completed successfully.
>> File Transfer Completed Successfully at: 2016-11-17
>> 04:30:51.569 QUIT
>> 221 Quit command received. Goodbye.
>>
>> This is what we are getting on the failures:
>>
>> 250 Transfer completed successfully.
>> 200 OK
>>
>> File Transfer Failed at: 2016-11-17 04:36:50.246 Error
>> Occurred Transmitting File to Remote System, aborting...
>>
>> Read timed out
>>
>> Here's the full log from the failed transfer:
>>
>> Using FTPS Connection with Protocol: TLS and Explicit Security
>> Request
>> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
>> 220 Connection will close if idle for more than 5 minutes.
>> AUTH TLS
>> 234 Security environment established - ready for negotiation
>> Connected to mvsp.xxx on xxxx PBSZ 0
>> 200 Protection buffer size accepted
>> PROT P
>> 200 Data connection protection set to private USER *******
>> 331 Send password please.
>> PASS *******
>> 230 C836PTS is logged on. Working directory is "C836PTS.".
>> PWD
>> 257 "'C836PTS.'" is working directory.
>>
>> Using a Packet Size of: 4096
>> Control KeepAliveTimeOut: 60
>> Control KeepAliveReplyTimeOut: 5000
>> FEAT
>> 211- Extensions supported
>> AUTH TLS
>> PBSZ
>> PROT
>> 211 End
>>
>> ...FTPClientUtil Connected Successfully!
>> SITE TRAIL
>> 200 SITE command was accepted
>> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
>> 200 SITE command was accepted
>> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
>> 200 SITE command was accepted
>> SITE CONDDISP=delete
>> 200 SITE command was accepted
>>
>>
>> Sending:
>> Local: d:\LargeExtracts\full.txt
>> Remote: 'A836.ORG.CAMA2.FULL'
>> Starting at: 2016-11-17 09:27:34.798 Total Bytes To
>> Send: 2101359588 PASV
>> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx) STOR
>> 'A836.ORG.CAMA2.FULL'
>> 125 Storing data set A836.ORG.CAMA2.FULL
>> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
>> 105070592
>> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
>> 210137088
>> ...
>> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
>> 2101359588
>> 250 Transfer completed successfully.
>> 200 OK
>>
>> File Transfer Failed at: 2016-11-17 09:32:26.127
>> Error Occurred Transmitting File to Remote System, aborting...
>>
>> Read timed out
>> java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(Unknown Source)
>> at java.net.SocketInputStream.read(Unknown Source)
>> at sun.security.ssl.InputRecord.readFully(Unknown Source)
>> at sun.security.ssl.InputRecord.read(Unknown Source)
>> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
>> Source)
>> at
>> sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
>> Source)
>> at sun.security.ssl.AppInputStream.read(Unknown Source)
>> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
>> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
>> at sun.nio.cs.StreamDecoder.read(Unknown Source)
>> at java.io.InputStreamReader.read(Unknown Source)
>> at java.io.BufferedReader.fill(Unknown Source)
>> at java.io.BufferedReader.readLine(Unknown Source)
>> at java.io.BufferedReader.readLine(Unknown Source)
>> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
>> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
>> java:304)
>> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
>> FTPClient.java:3697)
>> at org.apache.commons.net.ftp.FTPClient._storeFile(
>> FTPClient.java:669)
>> at org.apache.commons.net.ftp.FTPClient.__storeFile(
>> FTPClient.java:624)
>> at org.apache.commons.net.ftp.
>> FTPClient.storeFile(FTPClient.java:1976)
>> at gov.nyc.ftputils.FTPConnection.sendFile(
>> FTPConnection.java:695)
>> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:54)
>> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:96)
>> at
>> gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
>> FTPPropertyProcessor.java:126)
>> at gov.nyc.ftputils.FTPClientUtil.main(
>> FTPClientUtil.java:378)
>> FTPClientUtil has Failed
>> QUIT
>> 221 Quit command received. Goodbye.
>>
>>
>> Here's a code snippet of the transfer code:
>>
>> try{
>> if (trace){
>>
>> System.out.println("Total Bytes To Send: " + localFile.length());
>> CopyStreamListener
>> listener = new ProgressStreamListener(localFile.length());
>>
>> ftp.setCopyStreamListener( listener);
>> }
>>
>> InputStream input = new FileInputStream(localFileName);
>> boolean result = ftp.storeFile(tempRemoteFileName, input);
>> input.close();
>>
>> if (result){
>> System.out.println("\tFile Transfer
>> Completed Successfully at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>> ...
>> }
>> else
>> System.out.println("\tFile Transfer
>> Failed
>> at: " + sdf.format(Calendar.getInstance().getTime()));
>>
>> return result;
>> }catch (CopyStreamException cse){
>> System.err.println("\n\tFile Transfer Failed at: " +
>> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred
>> after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> cse.printStackTrace(System.err);
>> System.err.println("\n\nIOException Stack Trace that
>> Caused the Error:\n");
>> cse.getIOException().printStackTrace(System.err);
>> return false;
>> }catch (IOException ioe){
>> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> System.err.println(ioe.getMessage());
>> ioe.printStackTrace(System.err);
>> return false;
>> }catch (Exception e){
>> System.err.println("\n\tFile Transfer Failed at: " +
>> sdf.format(Calendar.getInstance().getTime()));
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> e.printStackTrace(System.err);
>> return false;
>> }
>> }
>>
>>
>> Thanks,
>>
>> Mike...
>>
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence
> with Hibernate, Second Edition
> <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygreg
> ory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1617290459>
> JUnit in Action, Second Edition
> <https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygreg
> ory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1935182021>
> Spring Batch in Action
> <https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7
> D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+i
> n+Action>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1935182951>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [net] FTPClient Transfer succeeds but fails in cleanUp
Posted by sebb <se...@gmail.com>.
Does the same code fail when sending the same file to a different host?
Have you tried using the example utility?
http://commons.apache.org/proper/commons-net/#Examples
This exercises most of the FTP methods.
On 17 November 2016 at 18:01, Seganti, Michael (DOF)
<Se...@finance.nyc.gov> wrote:
> I just updated to 3.5, and this file is consistently failing.
>
> I no longer receive the
>
> 250 Transfer completed successfully.
> 200 OK
>
> Replies.
>
> Log Snippet:
>
> 2016-11-17 12:56:33.870 - 100 Percent Complete, Total Bytes Transferred: 2101359588
>
> File Transfer Failed at: 2016-11-17 12:56:38.885
> Error Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at sun.security.ssl.InputRecord.readFully(Unknown Source)
> at sun.security.ssl.InputRecord.read(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> at sun.security.ssl.AppInputStream.read(Unknown Source)
> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.fill(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
> at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
> at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
> at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
> at gov.nyc.ftputils.FTPConnection.sendFile(FTPConnection.java:695)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:54)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:96)
> at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(FTPPropertyProcessor.java:126)
> at gov.nyc.ftputils.FTPClientUtil.main(FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> -----Original Message-----
> From: Seganti, Michael (DOF) [mailto:SegantiM@finance.nyc.gov]
> Sent: Thursday, November 17, 2016 12:34 PM
> To: Commons Users List
> Subject: RE: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> I need to do a quick test prior to pushing to production, but yes, that's my next step.
>
> I didn't see anything that stood out in the release notes. It would make life easier if this was reproduceable...
>
> -----Original Message-----
> From: Gary Gregory [mailto:garydgregory@gmail.com]
> Sent: Thursday, November 17, 2016 12:31 PM
> To: Commons Users List
> Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> Can you try version 3.5?
>
> Gary
>
> On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) < SegantiM@finance.nyc.gov> wrote:
>
>> I'm having a sporadic issue transferring files to a Mainframe, was
>> wondering if anyone else has run into this. I'm trying to figure out
>> if this is a Mainframe issue, or a net commons issue.
>>
>> I'm using Java 1.7 and commons-net-3.3.jar
>>
>> The file transfer completes, but it looks like either the 200 OK is
>> sent over the wrong connection, or is not properly being parsed and is
>> causing the cleanup to fail.
>>
>> According to the reply: IBM FTP CS V2R2 is the version where the issue
>> occurs.
>>
>> I have a second mainframe that is running IBM FTP CS V1R13 that
>> doesn't seem to have this issue
>>
>> In a normal transfer, you would see:
>>
>> 250 Transfer completed successfully.
>> File Transfer Completed Successfully at: 2016-11-17
>> 04:30:51.569 QUIT
>> 221 Quit command received. Goodbye.
>>
>> This is what we are getting on the failures:
>>
>> 250 Transfer completed successfully.
>> 200 OK
>>
>> File Transfer Failed at: 2016-11-17 04:36:50.246 Error Occurred
>> Transmitting File to Remote System, aborting...
>>
>> Read timed out
>>
>> Here's the full log from the failed transfer:
>>
>> Using FTPS Connection with Protocol: TLS and Explicit Security Request
>> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
>> 220 Connection will close if idle for more than 5 minutes.
>> AUTH TLS
>> 234 Security environment established - ready for negotiation Connected
>> to mvsp.xxx on xxxx PBSZ 0
>> 200 Protection buffer size accepted
>> PROT P
>> 200 Data connection protection set to private USER *******
>> 331 Send password please.
>> PASS *******
>> 230 C836PTS is logged on. Working directory is "C836PTS.".
>> PWD
>> 257 "'C836PTS.'" is working directory.
>>
>> Using a Packet Size of: 4096
>> Control KeepAliveTimeOut: 60
>> Control KeepAliveReplyTimeOut: 5000
>> FEAT
>> 211- Extensions supported
>> AUTH TLS
>> PBSZ
>> PROT
>> 211 End
>>
>> ...FTPClientUtil Connected Successfully!
>> SITE TRAIL
>> 200 SITE command was accepted
>> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
>> 200 SITE command was accepted
>> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
>> 200 SITE command was accepted
>> SITE CONDDISP=delete
>> 200 SITE command was accepted
>>
>>
>> Sending:
>> Local: d:\LargeExtracts\full.txt
>> Remote: 'A836.ORG.CAMA2.FULL'
>> Starting at: 2016-11-17 09:27:34.798 Total Bytes To
>> Send: 2101359588 PASV
>> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx) STOR
>> 'A836.ORG.CAMA2.FULL'
>> 125 Storing data set A836.ORG.CAMA2.FULL
>> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
>> 105070592
>> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
>> 210137088
>> ...
>> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
>> 2101359588
>> 250 Transfer completed successfully.
>> 200 OK
>>
>> File Transfer Failed at: 2016-11-17 09:32:26.127 Error
>> Occurred Transmitting File to Remote System, aborting...
>>
>> Read timed out
>> java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(Unknown Source)
>> at java.net.SocketInputStream.read(Unknown Source)
>> at sun.security.ssl.InputRecord.readFully(Unknown Source)
>> at sun.security.ssl.InputRecord.read(Unknown Source)
>> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
>> Source)
>> at
>> sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
>> Source)
>> at sun.security.ssl.AppInputStream.read(Unknown Source)
>> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
>> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
>> at sun.nio.cs.StreamDecoder.read(Unknown Source)
>> at java.io.InputStreamReader.read(Unknown Source)
>> at java.io.BufferedReader.fill(Unknown Source)
>> at java.io.BufferedReader.readLine(Unknown Source)
>> at java.io.BufferedReader.readLine(Unknown Source)
>> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
>> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
>> java:304)
>> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
>> FTPClient.java:3697)
>> at org.apache.commons.net.ftp.FTPClient._storeFile(
>> FTPClient.java:669)
>> at org.apache.commons.net.ftp.FTPClient.__storeFile(
>> FTPClient.java:624)
>> at org.apache.commons.net.ftp.
>> FTPClient.storeFile(FTPClient.java:1976)
>> at gov.nyc.ftputils.FTPConnection.sendFile(
>> FTPConnection.java:695)
>> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:54)
>> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:96)
>> at
>> gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
>> FTPPropertyProcessor.java:126)
>> at gov.nyc.ftputils.FTPClientUtil.main(
>> FTPClientUtil.java:378)
>> FTPClientUtil has Failed
>> QUIT
>> 221 Quit command received. Goodbye.
>>
>>
>> Here's a code snippet of the transfer code:
>>
>> try{
>> if (trace){
>>
>> System.out.println("Total Bytes To Send: " + localFile.length());
>> CopyStreamListener
>> listener = new ProgressStreamListener(localFile.length());
>>
>> ftp.setCopyStreamListener( listener);
>> }
>>
>> InputStream input = new FileInputStream(localFileName);
>> boolean result = ftp.storeFile(tempRemoteFileName, input);
>> input.close();
>>
>> if (result){
>> System.out.println("\tFile Transfer
>> Completed Successfully at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>> ...
>> }
>> else
>> System.out.println("\tFile Transfer
>> Failed
>> at: " + sdf.format(Calendar.getInstance().getTime()));
>>
>> return result;
>> }catch (CopyStreamException cse){
>> System.err.println("\n\tFile Transfer Failed at: " +
>> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred
>> after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> cse.printStackTrace(System.err);
>> System.err.println("\n\nIOException Stack Trace that
>> Caused the Error:\n");
>> cse.getIOException().printStackTrace(System.err);
>> return false;
>> }catch (IOException ioe){
>> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> System.err.println(ioe.getMessage());
>> ioe.printStackTrace(System.err);
>> return false;
>> }catch (Exception e){
>> System.err.println("\n\tFile Transfer Failed at: " +
>> sdf.format(Calendar.getInstance().getTime()));
>> System.err.println("Error Occurred Transmitting File
>> to Remote System, aborting...\n");
>> e.printStackTrace(System.err);
>> return false;
>> }
>> }
>>
>>
>> Thanks,
>>
>> Mike...
>>
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence with Hibernate, Second Edition <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
> JUnit in Action, Second Edition
> <https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
> Spring Batch in Action
> <https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: [net] FTPClient Transfer succeeds but fails in cleanUp
Posted by "Seganti, Michael (DOF)" <Se...@finance.nyc.gov>.
I just updated to 3.5, and this file is consistently failing.
I no longer receive the
250 Transfer completed successfully.
200 OK
Replies.
Log Snippet:
2016-11-17 12:56:33.870 - 100 Percent Complete, Total Bytes Transferred: 2101359588
File Transfer Failed at: 2016-11-17 12:56:38.885
Error Occurred Transmitting File to Remote System, aborting...
Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
at gov.nyc.ftputils.FTPConnection.sendFile(FTPConnection.java:695)
at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:54)
at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:96)
at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(FTPPropertyProcessor.java:126)
at gov.nyc.ftputils.FTPClientUtil.main(FTPClientUtil.java:378)
FTPClientUtil has Failed
QUIT
221 Quit command received. Goodbye.
-----Original Message-----
From: Seganti, Michael (DOF) [mailto:SegantiM@finance.nyc.gov]
Sent: Thursday, November 17, 2016 12:34 PM
To: Commons Users List
Subject: RE: [net] FTPClient Transfer succeeds but fails in cleanUp
I need to do a quick test prior to pushing to production, but yes, that's my next step.
I didn't see anything that stood out in the release notes. It would make life easier if this was reproduceable...
-----Original Message-----
From: Gary Gregory [mailto:garydgregory@gmail.com]
Sent: Thursday, November 17, 2016 12:31 PM
To: Commons Users List
Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
Can you try version 3.5?
Gary
On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) < SegantiM@finance.nyc.gov> wrote:
> I'm having a sporadic issue transferring files to a Mainframe, was
> wondering if anyone else has run into this. I'm trying to figure out
> if this is a Mainframe issue, or a net commons issue.
>
> I'm using Java 1.7 and commons-net-3.3.jar
>
> The file transfer completes, but it looks like either the 200 OK is
> sent over the wrong connection, or is not properly being parsed and is
> causing the cleanup to fail.
>
> According to the reply: IBM FTP CS V2R2 is the version where the issue
> occurs.
>
> I have a second mainframe that is running IBM FTP CS V1R13 that
> doesn't seem to have this issue
>
> In a normal transfer, you would see:
>
> 250 Transfer completed successfully.
> File Transfer Completed Successfully at: 2016-11-17
> 04:30:51.569 QUIT
> 221 Quit command received. Goodbye.
>
> This is what we are getting on the failures:
>
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 04:36:50.246 Error Occurred
> Transmitting File to Remote System, aborting...
>
> Read timed out
>
> Here's the full log from the failed transfer:
>
> Using FTPS Connection with Protocol: TLS and Explicit Security Request
> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
> 220 Connection will close if idle for more than 5 minutes.
> AUTH TLS
> 234 Security environment established - ready for negotiation Connected
> to mvsp.xxx on xxxx PBSZ 0
> 200 Protection buffer size accepted
> PROT P
> 200 Data connection protection set to private USER *******
> 331 Send password please.
> PASS *******
> 230 C836PTS is logged on. Working directory is "C836PTS.".
> PWD
> 257 "'C836PTS.'" is working directory.
>
> Using a Packet Size of: 4096
> Control KeepAliveTimeOut: 60
> Control KeepAliveReplyTimeOut: 5000
> FEAT
> 211- Extensions supported
> AUTH TLS
> PBSZ
> PROT
> 211 End
>
> ...FTPClientUtil Connected Successfully!
> SITE TRAIL
> 200 SITE command was accepted
> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
> 200 SITE command was accepted
> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
> 200 SITE command was accepted
> SITE CONDDISP=delete
> 200 SITE command was accepted
>
>
> Sending:
> Local: d:\LargeExtracts\full.txt
> Remote: 'A836.ORG.CAMA2.FULL'
> Starting at: 2016-11-17 09:27:34.798 Total Bytes To
> Send: 2101359588 PASV
> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx) STOR
> 'A836.ORG.CAMA2.FULL'
> 125 Storing data set A836.ORG.CAMA2.FULL
> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
> 105070592
> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
> 210137088
> ...
> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
> 2101359588
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 09:32:26.127 Error
> Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at sun.security.ssl.InputRecord.readFully(Unknown Source)
> at sun.security.ssl.InputRecord.read(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
> Source)
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
> Source)
> at sun.security.ssl.AppInputStream.read(Unknown Source)
> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.fill(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
> java:304)
> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
> FTPClient.java:3697)
> at org.apache.commons.net.ftp.FTPClient._storeFile(
> FTPClient.java:669)
> at org.apache.commons.net.ftp.FTPClient.__storeFile(
> FTPClient.java:624)
> at org.apache.commons.net.ftp.
> FTPClient.storeFile(FTPClient.java:1976)
> at gov.nyc.ftputils.FTPConnection.sendFile(
> FTPConnection.java:695)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:54)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:96)
> at
> gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
> FTPPropertyProcessor.java:126)
> at gov.nyc.ftputils.FTPClientUtil.main(
> FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> Here's a code snippet of the transfer code:
>
> try{
> if (trace){
>
> System.out.println("Total Bytes To Send: " + localFile.length());
> CopyStreamListener
> listener = new ProgressStreamListener(localFile.length());
>
> ftp.setCopyStreamListener( listener);
> }
>
> InputStream input = new FileInputStream(localFileName);
> boolean result = ftp.storeFile(tempRemoteFileName, input);
> input.close();
>
> if (result){
> System.out.println("\tFile Transfer
> Completed Successfully at: " + sdf.format(Calendar.
> getInstance().getTime()));
> ...
> }
> else
> System.out.println("\tFile Transfer
> Failed
> at: " + sdf.format(Calendar.getInstance().getTime()));
>
> return result;
> }catch (CopyStreamException cse){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred
> after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> cse.printStackTrace(System.err);
> System.err.println("\n\nIOException Stack Trace that
> Caused the Error:\n");
> cse.getIOException().printStackTrace(System.err);
> return false;
> }catch (IOException ioe){
> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
> getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> System.err.println(ioe.getMessage());
> ioe.printStackTrace(System.err);
> return false;
> }catch (Exception e){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> e.printStackTrace(System.err);
> return false;
> }
> }
>
>
> Thanks,
>
> Mike...
>
--
E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence with Hibernate, Second Edition <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
JUnit in Action, Second Edition
<https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
Spring Batch in Action
<https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
RE: [net] FTPClient Transfer succeeds but fails in cleanUp
Posted by "Seganti, Michael (DOF)" <Se...@finance.nyc.gov>.
I need to do a quick test prior to pushing to production, but yes, that's my next step.
I didn't see anything that stood out in the release notes. It would make life easier if this was reproduceable...
-----Original Message-----
From: Gary Gregory [mailto:garydgregory@gmail.com]
Sent: Thursday, November 17, 2016 12:31 PM
To: Commons Users List
Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
Can you try version 3.5?
Gary
On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) < SegantiM@finance.nyc.gov> wrote:
> I'm having a sporadic issue transferring files to a Mainframe, was
> wondering if anyone else has run into this. I'm trying to figure out
> if this is a Mainframe issue, or a net commons issue.
>
> I'm using Java 1.7 and commons-net-3.3.jar
>
> The file transfer completes, but it looks like either the 200 OK is
> sent over the wrong connection, or is not properly being parsed and is
> causing the cleanup to fail.
>
> According to the reply: IBM FTP CS V2R2 is the version where the issue
> occurs.
>
> I have a second mainframe that is running IBM FTP CS V1R13 that
> doesn't seem to have this issue
>
> In a normal transfer, you would see:
>
> 250 Transfer completed successfully.
> File Transfer Completed Successfully at: 2016-11-17
> 04:30:51.569 QUIT
> 221 Quit command received. Goodbye.
>
> This is what we are getting on the failures:
>
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 04:36:50.246 Error Occurred
> Transmitting File to Remote System, aborting...
>
> Read timed out
>
> Here's the full log from the failed transfer:
>
> Using FTPS Connection with Protocol: TLS and Explicit Security Request
> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
> 220 Connection will close if idle for more than 5 minutes.
> AUTH TLS
> 234 Security environment established - ready for negotiation Connected
> to mvsp.xxx on xxxx PBSZ 0
> 200 Protection buffer size accepted
> PROT P
> 200 Data connection protection set to private USER *******
> 331 Send password please.
> PASS *******
> 230 C836PTS is logged on. Working directory is "C836PTS.".
> PWD
> 257 "'C836PTS.'" is working directory.
>
> Using a Packet Size of: 4096
> Control KeepAliveTimeOut: 60
> Control KeepAliveReplyTimeOut: 5000
> FEAT
> 211- Extensions supported
> AUTH TLS
> PBSZ
> PROT
> 211 End
>
> ...FTPClientUtil Connected Successfully!
> SITE TRAIL
> 200 SITE command was accepted
> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
> 200 SITE command was accepted
> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
> 200 SITE command was accepted
> SITE CONDDISP=delete
> 200 SITE command was accepted
>
>
> Sending:
> Local: d:\LargeExtracts\full.txt
> Remote: 'A836.ORG.CAMA2.FULL'
> Starting at: 2016-11-17 09:27:34.798 Total Bytes To
> Send: 2101359588 PASV
> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx) STOR
> 'A836.ORG.CAMA2.FULL'
> 125 Storing data set A836.ORG.CAMA2.FULL
> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
> 105070592
> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
> 210137088
> ...
> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
> 2101359588
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 09:32:26.127 Error
> Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at sun.security.ssl.InputRecord.readFully(Unknown Source)
> at sun.security.ssl.InputRecord.read(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
> Source)
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
> Source)
> at sun.security.ssl.AppInputStream.read(Unknown Source)
> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.fill(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
> java:304)
> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
> FTPClient.java:3697)
> at org.apache.commons.net.ftp.FTPClient._storeFile(
> FTPClient.java:669)
> at org.apache.commons.net.ftp.FTPClient.__storeFile(
> FTPClient.java:624)
> at org.apache.commons.net.ftp.
> FTPClient.storeFile(FTPClient.java:1976)
> at gov.nyc.ftputils.FTPConnection.sendFile(
> FTPConnection.java:695)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:54)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:96)
> at
> gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
> FTPPropertyProcessor.java:126)
> at gov.nyc.ftputils.FTPClientUtil.main(
> FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> Here's a code snippet of the transfer code:
>
> try{
> if (trace){
>
> System.out.println("Total Bytes To Send: " + localFile.length());
> CopyStreamListener
> listener = new ProgressStreamListener(localFile.length());
>
> ftp.setCopyStreamListener( listener);
> }
>
> InputStream input = new FileInputStream(localFileName);
> boolean result = ftp.storeFile(tempRemoteFileName, input);
> input.close();
>
> if (result){
> System.out.println("\tFile Transfer
> Completed Successfully at: " + sdf.format(Calendar.
> getInstance().getTime()));
> ...
> }
> else
> System.out.println("\tFile Transfer
> Failed
> at: " + sdf.format(Calendar.getInstance().getTime()));
>
> return result;
> }catch (CopyStreamException cse){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred
> after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> cse.printStackTrace(System.err);
> System.err.println("\n\nIOException Stack Trace that
> Caused the Error:\n");
> cse.getIOException().printStackTrace(System.err);
> return false;
> }catch (IOException ioe){
> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
> getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> System.err.println(ioe.getMessage());
> ioe.printStackTrace(System.err);
> return false;
> }catch (Exception e){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File
> to Remote System, aborting...\n");
> e.printStackTrace(System.err);
> return false;
> }
> }
>
>
> Thanks,
>
> Mike...
>
--
E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence with Hibernate, Second Edition <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
JUnit in Action, Second Edition
<https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
Spring Batch in Action
<https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
Re: [net] FTPClient Transfer succeeds but fails in cleanUp
Posted by Gary Gregory <ga...@gmail.com>.
Can you try version 3.5?
Gary
On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) <
SegantiM@finance.nyc.gov> wrote:
> I'm having a sporadic issue transferring files to a Mainframe, was
> wondering if anyone else has run into this. I'm trying to figure out if
> this is a Mainframe issue, or a net commons issue.
>
> I'm using Java 1.7 and commons-net-3.3.jar
>
> The file transfer completes, but it looks like either the 200 OK is sent
> over the wrong connection, or is not properly being parsed and is causing
> the cleanup to fail.
>
> According to the reply: IBM FTP CS V2R2 is the version where the issue
> occurs.
>
> I have a second mainframe that is running IBM FTP CS V1R13 that doesn't
> seem to have this issue
>
> In a normal transfer, you would see:
>
> 250 Transfer completed successfully.
> File Transfer Completed Successfully at: 2016-11-17 04:30:51.569
> QUIT
> 221 Quit command received. Goodbye.
>
> This is what we are getting on the failures:
>
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 04:36:50.246
> Error Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
>
> Here's the full log from the failed transfer:
>
> Using FTPS Connection with Protocol: TLS and Explicit Security Request
> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
> 220 Connection will close if idle for more than 5 minutes.
> AUTH TLS
> 234 Security environment established - ready for negotiation
> Connected to mvsp.xxx on xxxx
> PBSZ 0
> 200 Protection buffer size accepted
> PROT P
> 200 Data connection protection set to private
> USER *******
> 331 Send password please.
> PASS *******
> 230 C836PTS is logged on. Working directory is "C836PTS.".
> PWD
> 257 "'C836PTS.'" is working directory.
>
> Using a Packet Size of: 4096
> Control KeepAliveTimeOut: 60
> Control KeepAliveReplyTimeOut: 5000
> FEAT
> 211- Extensions supported
> AUTH TLS
> PBSZ
> PROT
> 211 End
>
> ...FTPClientUtil Connected Successfully!
> SITE TRAIL
> 200 SITE command was accepted
> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
> 200 SITE command was accepted
> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
> 200 SITE command was accepted
> SITE CONDDISP=delete
> 200 SITE command was accepted
>
>
> Sending:
> Local: d:\LargeExtracts\full.txt
> Remote: 'A836.ORG.CAMA2.FULL'
> Starting at: 2016-11-17 09:27:34.798
> Total Bytes To Send: 2101359588
> PASV
> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx)
> STOR 'A836.ORG.CAMA2.FULL'
> 125 Storing data set A836.ORG.CAMA2.FULL
> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
> 105070592
> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
> 210137088
> ...
> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
> 2101359588
> 250 Transfer completed successfully.
> 200 OK
>
> File Transfer Failed at: 2016-11-17 09:32:26.127
> Error Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(Unknown Source)
> at java.net.SocketInputStream.read(Unknown Source)
> at sun.security.ssl.InputRecord.readFully(Unknown Source)
> at sun.security.ssl.InputRecord.read(Unknown Source)
> at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
> Source)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
> Source)
> at sun.security.ssl.AppInputStream.read(Unknown Source)
> at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
> at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
> at sun.nio.cs.StreamDecoder.read(Unknown Source)
> at java.io.InputStreamReader.read(Unknown Source)
> at java.io.BufferedReader.fill(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at java.io.BufferedReader.readLine(Unknown Source)
> at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
> at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
> java:304)
> at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
> FTPClient.java:3697)
> at org.apache.commons.net.ftp.FTPClient._storeFile(
> FTPClient.java:669)
> at org.apache.commons.net.ftp.FTPClient.__storeFile(
> FTPClient.java:624)
> at org.apache.commons.net.ftp.
> FTPClient.storeFile(FTPClient.java:1976)
> at gov.nyc.ftputils.FTPConnection.sendFile(
> FTPConnection.java:695)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:54)
> at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
> FTPPropertyProcessor.java:96)
> at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
> FTPPropertyProcessor.java:126)
> at gov.nyc.ftputils.FTPClientUtil.main(
> FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> Here's a code snippet of the transfer code:
>
> try{
> if (trace){
> System.out.println("Total
> Bytes To Send: " + localFile.length());
> CopyStreamListener
> listener = new ProgressStreamListener(localFile.length());
> ftp.setCopyStreamListener(
> listener);
> }
>
> InputStream input = new FileInputStream(localFileName);
> boolean result = ftp.storeFile(tempRemoteFileName, input);
> input.close();
>
> if (result){
> System.out.println("\tFile Transfer
> Completed Successfully at: " + sdf.format(Calendar.
> getInstance().getTime()));
> ...
> }
> else
> System.out.println("\tFile Transfer Failed
> at: " + sdf.format(Calendar.getInstance().getTime()));
>
> return result;
> }catch (CopyStreamException cse){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred after
> " + cse.getTotalBytesTransferred() + " Bytes Transferred");
> System.err.println("Error Occurred Transmitting File to
> Remote System, aborting...\n");
> cse.printStackTrace(System.err);
> System.err.println("\n\nIOException Stack Trace that
> Caused the Error:\n");
> cse.getIOException().printStackTrace(System.err);
> return false;
> }catch (IOException ioe){
> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
> getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File to
> Remote System, aborting...\n");
> System.err.println(ioe.getMessage());
> ioe.printStackTrace(System.err);
> return false;
> }catch (Exception e){
> System.err.println("\n\tFile Transfer Failed at: " +
> sdf.format(Calendar.getInstance().getTime()));
> System.err.println("Error Occurred Transmitting File to
> Remote System, aborting...\n");
> e.printStackTrace(System.err);
> return false;
> }
> }
>
>
> Thanks,
>
> Mike...
>
--
E-Mail: garydgregory@gmail.com | ggregory@apache.org
Java Persistence with Hibernate, Second Edition
<https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
JUnit in Action, Second Edition
<https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
Spring Batch in Action
<https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory