You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Martin Stockhammer (Created) (JIRA)" <ji...@apache.org> on 2012/04/19 22:58:40 UTC
[jira] [Created] (VFS-410) If using getInputStream(long position)
on SftpFileObject the file is read into memory
If using getInputStream(long position) on SftpFileObject the file is read into memory
-------------------------------------------------------------------------------------
Key: VFS-410
URL: https://issues.apache.org/jira/browse/VFS-410
Project: Commons VFS
Issue Type: Bug
Affects Versions: 2.0
Reporter: Martin Stockhammer
Fix For: 2.1
The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
Newer versions of JSch provide a method
{{channel.get(file, monitor, filePointer);}}
that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-410) If using getInputStream(long position)
on SftpFileObject the file is read into memory
Posted by "Martin Stockhammer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Stockhammer updated VFS-410:
-----------------------------------
Attachment: SftpFileObject.java.patch
Ok, I removed the version check and attached the patch as file with license grant. I hope it has the correct format.
> If using getInputStream(long position) on SftpFileObject the file is read into memory
> -------------------------------------------------------------------------------------
>
> Key: VFS-410
> URL: https://issues.apache.org/jira/browse/VFS-410
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0
> Reporter: Martin Stockhammer
> Fix For: 2.1
>
> Attachments: SftpFileObject.java.patch
>
>
> The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
> Newer versions of JSch provide a method
> {{channel.get(file, monitor, filePointer);}}
> that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-410) If using getInputStream(long position)
on SftpFileObject the file is read into memory
Posted by "Gary D. Gregory (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257970#comment-13257970 ]
Gary D. Gregory commented on VFS-410:
-------------------------------------
Hi Martin,
In order to use a patch, it must be attached to this issue as a patch file and granted the Apache license.
I would not bother with the version check, the POM already specifies 0.1.46 and I am testing a build with 0.1.47 as we speak...
Thank you,
Gary
> If using getInputStream(long position) on SftpFileObject the file is read into memory
> -------------------------------------------------------------------------------------
>
> Key: VFS-410
> URL: https://issues.apache.org/jira/browse/VFS-410
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0
> Reporter: Martin Stockhammer
> Fix For: 2.1
>
>
> The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
> Newer versions of JSch provide a method
> {{channel.get(file, monitor, filePointer);}}
> that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (VFS-410) [SFTP] SftpFileObject
getInputStream(long) reads the whole file into memory
Posted by "Gary D. Gregory (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary D. Gregory updated VFS-410:
--------------------------------
Summary: [SFTP] SftpFileObject getInputStream(long) reads the whole file into memory (was: If using getInputStream(long position) on SftpFileObject the file is read into memory)
> [SFTP] SftpFileObject getInputStream(long) reads the whole file into memory
> ---------------------------------------------------------------------------
>
> Key: VFS-410
> URL: https://issues.apache.org/jira/browse/VFS-410
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0
> Reporter: Martin Stockhammer
> Fix For: 2.1
>
> Attachments: SftpFileObject.java.patch
>
>
> The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
> Newer versions of JSch provide a method
> {{channel.get(file, monitor, filePointer);}}
> that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (VFS-410) If using getInputStream(long position)
on SftpFileObject the file is read into memory
Posted by "Martin Stockhammer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257812#comment-13257812 ]
Martin Stockhammer commented on VFS-410:
----------------------------------------
Here is a patch that uses the new method. I added a version check for JSch to be sure the method is allowed.
{code}
Index: src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileObject.java
===================================================================
--- src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileObject.java (revision 1327822)
+++ src/main/java/org/apache/commons/vfs2/provider/sftp/SftpFileObject.java (working copy)
@@ -42,6 +42,7 @@
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.ChannelSftp.LsEntry;
+import com.jcraft.jsch.JSch;
import com.jcraft.jsch.SftpATTRS;
import com.jcraft.jsch.SftpException;
@@ -53,7 +54,25 @@
public class SftpFileObject extends AbstractFileObject
{
private static final long MOD_TIME_FACTOR = 1000L;
+ public static boolean useInputStream = false;
+ // Checking for JSch version. Versions >= 0.1.46 use the InputStream from
+ // the SftpChannel.
+ static {
+ try {
+ String[] versions = JSch.class.getDeclaredField("VERSION")
+ .get(null).toString().split("\\.");
+ if (versions.length == 3) {
+ int[] vers = { Integer.parseInt(versions[0]),
+ Integer.parseInt(versions[1]),
+ Integer.parseInt(versions[2]) };
+ useInputStream = vers[0] > 0 || vers[1] > 1
+ || (vers[1] == 1 && vers[2] >= 46);
+ }
+ } catch (Exception e) {
+ System.err.println("Retrieval of JSch version failed");
+ }
+ }
private final SftpFileSystem fileSystem;
private SftpATTRS attrs;
private final String relPath;
@@ -447,32 +466,39 @@
/**
* Creates an input stream to read the file content from.
+ * The input stream is starting at the given position in the file.
*/
InputStream getInputStream(long filePointer) throws IOException
{
final ChannelSftp channel = fileSystem.getChannel();
- try
- {
- // hmmm - using the in memory method is soooo much faster ...
- // TODO - Don't read the entire file into memory. Use the
- // stream-based methods on ChannelSftp once they work properly final
- // .... no stream based method with resume???
- ByteArrayOutputStream outstr = new ByteArrayOutputStream();
- try
- {
- channel.get(getName().getPathDecoded(), outstr, null,
- ChannelSftp.RESUME, filePointer);
- }
- catch (SftpException e)
- {
+ if (useInputStream) {
+ // Using InputStream directly from the channel.
+ // Its much faster than the memory method.
+ try {
+ InputStream is = channel.get(getName().getPathDecoded(), null,
+ filePointer);
+ return new SftpInputStream(channel, is);
+ } catch (SftpException e) {
+ fileSystem.putChannel(channel);
throw new FileSystemException(e);
+ }
+
+ } else {
+ try {
+ // Reading completely into the memory.
+ ByteArrayOutputStream outstr = new ByteArrayOutputStream();
+ try {
+ channel.get(getName().getPathDecoded(), outstr, null,
+ ChannelSftp.RESUME, filePointer);
+ } catch (SftpException e) {
+ throw new FileSystemException(e);
+ }
+ outstr.close();
+ return new ByteArrayInputStream(outstr.toByteArray());
+
+ } finally {
+ fileSystem.putChannel(channel);
}
- outstr.close();
- return new ByteArrayInputStream(outstr.toByteArray());
- }
- finally
- {
- fileSystem.putChannel(channel);
}
}
{code}
> If using getInputStream(long position) on SftpFileObject the file is read into memory
> -------------------------------------------------------------------------------------
>
> Key: VFS-410
> URL: https://issues.apache.org/jira/browse/VFS-410
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0
> Reporter: Martin Stockhammer
> Fix For: 2.1
>
>
> The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
> Newer versions of JSch provide a method
> {{channel.get(file, monitor, filePointer);}}
> that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (VFS-410) [SFTP] SftpFileObject
getInputStream(long) reads the whole file into memory
Posted by "Gary D. Gregory (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary D. Gregory resolved VFS-410.
---------------------------------
Resolution: Fixed
Martin,
Thank you for providing the patch.
It has been applied with only slight changes.
Committed revision 1328346.
Gary
> [SFTP] SftpFileObject getInputStream(long) reads the whole file into memory
> ---------------------------------------------------------------------------
>
> Key: VFS-410
> URL: https://issues.apache.org/jira/browse/VFS-410
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0
> Reporter: Martin Stockhammer
> Fix For: 2.1
>
> Attachments: SftpFileObject.java.patch
>
>
> The method {{getInputStream(long filePointer)}} in {{SftpFileObject.java}} reads the complete file into memory to create the input stream.
> Newer versions of JSch provide a method
> {{channel.get(file, monitor, filePointer);}}
> that is much faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira