You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Michael Manion (JIRA)" <ji...@apache.org> on 2008/02/28 14:13:51 UTC
[jira] Created: (VFS-200) sftp fails when files are very large
sftp fails when files are very large
------------------------------------
Key: VFS-200
URL: https://issues.apache.org/jira/browse/VFS-200
Project: Commons VFS
Issue Type: Bug
Affects Versions: 1.0
Environment: Java EE 5
Reporter: Michael Manion
Priority: Blocker
We only get this error with very large files. We are using the latest release of JSch (0.1.37).
org.apache.commons.vfs.FileSystemException: Could not read file "sftp://[useridRemoved]@[ipAddrRemoved]/returned/ccp_allusers.20080219010439.01.noe.clean_C_.txt".
at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1052)
at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:312)
at com.ibm.account.batch.jobs.gecs.SecureFTPHelper.retrieveGECSResultsFiles(SecureFTPHelper.java:155)
Caused by: 4:
at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java(Compiled Code))
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:855)
at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:834)
at org.apache.commons.vfs.provider.sftp.SftpFileObject.doGetInputStream(SftpFileObject.java:362)
at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1048)
... 14 more
Caused by: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java(Inlined Compiled Code))
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java(Compiled Code))
... 19 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (VFS-200) sftp fails when files are very large
Posted by "Michael Manion (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581996#action_12581996 ]
Michael Manion commented on VFS-200:
------------------------------------
Maybe it would help if I provide more information on this issue. We are using Commons-VFS to login to an sftp server and download very 2 or 3 large files (several hundred MB each) every week. The download fails consistently with the error message seen in my original problem description. We have tested with smaller files and do not see the error. We are using the official 1.0 version of the Commons-VFS jarfile, along with the latest JSch jarfile (0.1.37).
We tried breaking one of the large files up into 20MB "chunks" on the remote sftp server. The result was that the first 4 of these 20MB files would download correctly, and then we get the same error while downloading the 5th one. Again, this is a consistent result.
> sftp fails when files are very large
> ------------------------------------
>
> Key: VFS-200
> URL: https://issues.apache.org/jira/browse/VFS-200
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 1.0
> Environment: Java EE 5
> Reporter: Michael Manion
> Priority: Blocker
>
> We only get this error with very large files. We are using the latest release of JSch (0.1.37).
> org.apache.commons.vfs.FileSystemException: Could not read file "sftp://[useridRemoved]@[ipAddrRemoved]/returned/ccp_allusers.20080219010439.01.noe.clean_C_.txt".
> at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1052)
> at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:312)
> at com.ibm.account.batch.jobs.gecs.SecureFTPHelper.retrieveGECSResultsFiles(SecureFTPHelper.java:155)
> Caused by: 4:
> at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java(Compiled Code))
> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:855)
> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:834)
> at org.apache.commons.vfs.provider.sftp.SftpFileObject.doGetInputStream(SftpFileObject.java:362)
> at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1048)
> ... 14 more
> Caused by: java.io.IOException: inputstream is closed
> at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java(Inlined Compiled Code))
> at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java(Compiled Code))
> ... 19 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (VFS-200) sftp fails when files are very large
Posted by "Trey Massingill (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637521#action_12637521 ]
Trey Massingill commented on VFS-200:
-------------------------------------
I think this is happening because the sftp provider for VFS is calling JSch methods that read the entire contents of the remote file into memory before providing the caller an input stream. In my situation, this caused an out of memory error.
The fix for this is for the VFS sftp provider to call the JSch method that return InputStream (and OutputStream) objects.
> sftp fails when files are very large
> ------------------------------------
>
> Key: VFS-200
> URL: https://issues.apache.org/jira/browse/VFS-200
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 1.0
> Environment: Java EE 5
> Reporter: Michael Manion
> Priority: Blocker
>
> We only get this error with very large files. We are using the latest release of JSch (0.1.37).
> org.apache.commons.vfs.FileSystemException: Could not read file "sftp://[useridRemoved]@[ipAddrRemoved]/returned/ccp_allusers.20080219010439.01.noe.clean_C_.txt".
> at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1052)
> at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:312)
> at com.ibm.account.batch.jobs.gecs.SecureFTPHelper.retrieveGECSResultsFiles(SecureFTPHelper.java:155)
> Caused by: 4:
> at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java(Compiled Code))
> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:855)
> at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:834)
> at org.apache.commons.vfs.provider.sftp.SftpFileObject.doGetInputStream(SftpFileObject.java:362)
> at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1048)
> ... 14 more
> Caused by: java.io.IOException: inputstream is closed
> at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java(Inlined Compiled Code))
> at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java(Compiled Code))
> ... 19 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.