You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "Marty Lamb (JIRA)" <ji...@apache.org> on 2007/01/15 18:30:27 UTC
[jira] Created: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
large Sftp transfers fail with OutOfMemoryError: Java heap space
----------------------------------------------------------------
Key: VFS-107
URL: https://issues.apache.org/jira/browse/VFS-107
Project: Commons VFS
Issue Type: Bug
Affects Versions: Nightly Builds
Environment: java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
Reporter: Marty Lamb
Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
This is noted in the source for SftpFileObject:
protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
{
// TODO - Don't write the entire file into memory. Use the stream-based
// methods on ChannelSftp once the work properly
final ChannelSftp channel = fileSystem.getChannel();
return new SftpOutputStream(channel);
}
although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Marty Lamb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479298 ]
Marty Lamb commented on VFS-107:
--------------------------------
Sorry, should have commented on this much sooner. That seems to work for me - thanks!
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
> Attachments: SftpFileObject.java
>
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Updated: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Stefan Risto (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Risto updated VFS-107:
-----------------------------
Attachment: SftpFileObject.java
I attached the patched SFTPFileObject java class, that seems to be working without buffering the whole file using the latest jsch lib.
The class is based on the VFS-1.0 source code. I only patched the buggy buffer part. If you are interested in what lines of code have changed, please diff it against the same file from VFS-1.0.
Hopefully this patch finds its way to VFS 1.1.
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
> Attachments: SftpFileObject.java
>
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Andrew Serff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479501 ]
Andrew Serff commented on VFS-107:
----------------------------------
Very cool. Thanks! Anyone know when v1.1 is slated to be released? I'm guess I'm going to be using the patched version for a little while.
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
> Attachments: SftpFileObject.java
>
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Resolved: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mario Ivankovits resolved VFS-107.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.1
ok, I've changed VFS head now to use the stream based methods, the test cases still work, so it looks like it works now.
Thanks!
Ciao,
Mario
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
> Fix For: 1.1
>
> Attachments: SftpFileObject.java
>
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Andrew Serff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12479211 ]
Andrew Serff commented on VFS-107:
----------------------------------
Any new news on this? I really need this one fixed and soon. If you point me to some documentation for the stream stuff you talk about, maybe I can take a stab at it. Let me know.
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Stefan Risto (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465187 ]
Stefan Risto commented on VFS-107:
----------------------------------
I don't know when you had the last try with jsch, but I read the change log of the latest version (0.1.30) and some of the fixed bugs might be related to your former problems. So please have a look at it.
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-107) large Sftp transfers fail with
OutOfMemoryError: Java heap space
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465113 ]
Mario Ivankovits commented on VFS-107:
--------------------------------------
Hi!
The stream-based methods were dog slow in the past. And when I say dog slow, I mean they were "double dog slow" - to an unusable extent ;-)
As far as I remember it had something to do with the piped streams jsch has to use internally in case of stream based operations.
I dont know if this has been fixed now, should give it another try.
> large Sftp transfers fail with OutOfMemoryError: Java heap space
> ----------------------------------------------------------------
>
> Key: VFS-107
> URL: https://issues.apache.org/jira/browse/VFS-107
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: java version "1.5.0_04"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)
> Linux version 2.6.17-1.2142_FC4 (bhcompile@hs20-bc1-4.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 4.0.2-8)) #1 Tue Jul 11 22:41:14 EDT 2006
> Reporter: Marty Lamb
>
> Calling SftpFileObject.getOutputStream() returns a descendant of ByteArrayOutputStream; nothing is written to the remote sftp server until the OutputStream is closed. For large data transfers, this exhausts local resources.
> This is noted in the source for SftpFileObject:
> protected OutputStream doGetOutputStream(boolean bAppend) throws Exception
> {
> // TODO - Don't write the entire file into memory. Use the stream-based
> // methods on ChannelSftp once the work properly
> final ChannelSftp channel = fileSystem.getChannel();
> return new SftpOutputStream(channel);
> }
> although it is not clear what "once the[y] work properly" is referring to.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org