You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Michael Melhem <mi...@managesoft.com> on 2005/06/08 14:29:05 UTC
[VFS] OutOfMemory Exception on large files
Hi Guys,
We have a problem with the DefaultFileContent-OutputStream upon handling
of large files. We get an OutOFMemory Exception on any upload of a file
> than approx 13 Meg, but never happens on the download (ie
inputstream) of the same file.
The Exception happens with the latest VFS code from svn, plus all
earlier snap shots I have tried.
We are using HTTP protocol. We have code something like this (in a
simple junit test):-
...
final FileObject toDownload = fsManager.resolveFile(sourceURI.toString());
final InputStream is = toDownload.getContent().getInputStream();
final FileObject toUpload = fsManager.resolveFile(targetURI.toString());
final OutPutStream os = toUpload.getContent().getOutputStream();
org.apache.commons.io.IOUtils.copy(is, os);
os.close() // this close will always cause an OutOfMemory Exception
(when file is approx larger than 13 M on our system)
is.close()
...
Note we use the commons.io.IOUtils for the copy which handles
stream-copy using 4k buffers AFIAK, so i dont think the problem is there.
Any help on this will be much appreciated!!
Regards,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] OutOfMemory Exception on large files
Posted by Torsten Curdt <tc...@apache.org>.
>> PS: shall we post the patch to bugzilla?
>>
>
> You mean the Piped version - sure, please ;-)
Well... :)
> But please post the temp-file workaround too, with low-priority and as
> enhancement - will see what the future brings.
Ok, workaround is filed. Michael is looking
into piped version as well.
cheers
--
Torsten
Re: [VFS] OutOfMemory Exception on large files
Posted by Matt Benson <gu...@yahoo.com>.
--- Mario Ivankovits <ma...@ops.co.at> wrote:
> Hi Torsten!
> >Michael came up with a workaround. Instead of
> >using a ByteArrayOutputStream we use a temp
> >file and a FileOutputStream.
> Nice idea!
>
> >He is currently looking into a Piped version.
> >Or what do you think how should we solve this?
> >
> I think a Piped version is the correct solution.
>
> But please have a look at the perfomance also.
> As I said the jdk Piped*Stream use an internal
> buffer of 1k. When I try
> to use the piped version in jsch (sftp) it gets dog
> slow.
> I blamed the JDK implementation of the pipe for
> this, though I havent
> investigated into this any further.
> But maybe we also nead our own implementation of
> Pipe-Streams.
FYI it looks a little hacky perhaps but Ant's codebase
has org.apache.tools.ant.util.LeadPipeInputStream that
a) doesn't die just because its writing Thread has
died, and b) allows you to specify (or change while in
use) the buffer size.
-Matt
__________________________________
Discover Yahoo!
Have fun online with music videos, cool games, IM and more. Check it out!
http://discover.yahoo.com/online.html
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] OutOfMemory Exception on large files
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi Torsten!
>Michael came up with a workaround. Instead of
>using a ByteArrayOutputStream we use a temp
>file and a FileOutputStream.
Nice idea!
>He is currently looking into a Piped version.
>Or what do you think how should we solve this?
>
I think a Piped version is the correct solution.
But please have a look at the perfomance also.
As I said the jdk Piped*Stream use an internal buffer of 1k. When I try
to use the piped version in jsch (sftp) it gets dog slow.
I blamed the JDK implementation of the pipe for this, though I havent
investigated into this any further.
But maybe we also nead our own implementation of Pipe-Streams.
>PS: shall we post the patch to bugzilla?
>
You mean the Piped version - sure, please ;-)
But please post the temp-file workaround too, with low-priority and as
enhancement - will see what the future brings.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [VFS] OutOfMemory Exception on large files
Posted by Torsten Curdt <tc...@apache.org>.
>> Ok ...thanks for the pointers - will see what I can do
>>
>
> Feel free to contact me if there is something else I can help with.
Thanks, mate
Michael came up with a workaround. Instead of
using a ByteArrayOutputStream we use a temp
file and a FileOutputStream. Of course is not
really that great because of the io overhead
but at least it can cope with big files.
He is currently looking into a Piped version.
Or what do you think how should we solve this?
cheers
--
Torsten
PS: shall we post the patch to bugzilla?
Re: [VFS] OutOfMemory Exception on large files
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi Torsten!
>Doh! ...
No problem!
>Ok ...thanks for the pointers - will see what I can do
>
Feel free to contact me if there is something else I can help with.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [VFS] OutOfMemory Exception on large files
Posted by Torsten Curdt <tc...@apache.org>.
>> We can even reproduce it with the "file:" protocol!
>>
>
> What? I cant belief.
Doh! ...forget what I said - all cool with "file:"
(obviously my jetlag is kickin in) Sorry for the
noise and thanks for the crosscheck.
>> Can you point me into the right direction ...haven't had
>> a closer look into the vfs codebase yet.
>>
>
> Please find:
> org.apache.commons.vfs.provider.webdav.WebdavFileObject and its inner
> class WebdavOutputStream
>
> Also the sftp FileProvider uses a byte buffer. Its easy to change this
> to jsch own "getOutputStream()" method but jsch internally also uses a
> pipe and it is dog slow then. Java uses a buffer of 1K for its pipe wich
> seems to be a huge performance bottleneck.
Ok ...thanks for the pointers - will see what I can do
cheers
--
Torsten
Re: [VFS] OutOfMemory Exception on large files
Posted by Mario Ivankovits <ma...@ops.co.at>.
Torsten Curdt wrote:
>>>We are using HTTP protocol. We have code something like this (in a
>>>simple junit test):-
>>>
>>Neither "http" nor "url" supports output streams, so I guess your target
>>URL is a "webdav://" resource, isnt it?
>>
>
>We can even reproduce it with the "file:" protocol!
>
What? I cant belief.
I tried:
FileSystemManager fsm = VFS.getManager();
FileObject foSrc = fsm.resolveFile("/home/im/vmware/vfstest.tbz2");
FileObject foDest =
fsm.resolveFile("/home/im/vmware/vfstest.tmp.tbz2");
foDest.copyFrom(foSrc, Selectors.SELECT_SELF);
FileObject foDest2 =
fsm.resolveFile("/home/im/vmware/vfstest.tmp2.tbz2");
InputStream is = foSrc.getContent().getInputStream();
OutputStream os = foDest2.getContent().getOutputStream();
CopyUtils.copy(is, os);
os.close();
is.close();
where vfstest.tbz2 is a 159.726.161 bytes big file. - works like a charme.
As you might see, I tried VFS own copyFrom and IOUtils, but ...
> org.apache.commons.io.IOUtils.copy(is, os);
>
... which version of IOUtils do you use?
I downloaded 1.0 but instead of finding a IOUtils.copy I found
CopyUtils.copy ...
>>Maybe it is possible to use Piped-Streams to pass through to the webdav
>>server, but currently I do not have enough spare time to do this -
>>contributions are welcome :-)
>>
>
>Can you point me into the right direction ...haven't had
>a closer look into the vfs codebase yet.
>
Please find:
org.apache.commons.vfs.provider.webdav.WebdavFileObject and its inner
class WebdavOutputStream
Also the sftp FileProvider uses a byte buffer. Its easy to change this
to jsch own "getOutputStream()" method but jsch internally also uses a
pipe and it is dog slow then. Java uses a buffer of 1K for its pipe wich
seems to be a huge performance bottleneck.
---
Mario
Re: [VFS] OutOfMemory Exception on large files
Posted by Torsten Curdt <tc...@apache.org>.
>> We are using HTTP protocol. We have code something like this (in a
>> simple junit test):-
>
> Neither "http" nor "url" supports output streams, so I guess your target
> URL is a "webdav://" resource, isnt it?
We can even reproduce it with the "file:" protocol!
final String sourceURI = "file:/Users/tcurdt/90MB";
final String targetURI = "file:/Users/tcurdt/test";
final FileSystemManager fsManager = VFS.getManager();
final FileObject toDownload = fsManager.resolveFile(sourceURI);
final InputStream is = toDownload.getContent().getInputStream();
final FileObject toUpload = fsManager.resolveFile(targetURI);
final OutputStream os = toUpload.getContent().getOutputStream();
org.apache.commons.io.IOUtils.copy(is, os);
os.close();
is.close();
> webdav buffers the output until close and send the whole stuff at once.
Uh!
> Maybe it is possible to use Piped-Streams to pass through to the webdav
> server, but currently I do not have enough spare time to do this -
> contributions are welcome :-)
Can you point me into the right direction ...haven't had
a closer look into the vfs codebase yet.
> As a workaround you could add "-Xmx512m" (or what ever fits your needs)
> to your JVM startup to allow java to allocate more memory.
...naaa ...that's not even a workaround
we are talking about biiiiiig files! :-)
cheers
--
Torsten
Re: [VFS] OutOfMemory Exception on large files
Posted by Mario Ivankovits <ma...@ops.co.at>.
Michael Melhem wrote:
> We are using HTTP protocol. We have code something like this (in a
> simple junit test):-
Neither "http" nor "url" supports output streams, so I guess your target
URL is a "webdav://" resource, isnt it?
webdav buffers the output until close and send the whole stuff at once.
Maybe it is possible to use Piped-Streams to pass through to the webdav
server, but currently I do not have enough spare time to do this -
contributions are welcome :-)
As a workaround you could add "-Xmx512m" (or what ever fits your needs)
to your JVM startup to allow java to allocate more memory.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org