You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by zizou77 <pa...@laposte.net> on 2006/10/02 16:29:00 UTC
OutOfMemoryError with VFS
Hi,
i would like to copy big big file (10G or 20G) with vfs.
So i use:
FileObject file = flowRepository.resolveFile("src.txt");
InputStream is = file.getContent().getInputStream();
FileObject flow = flowRepository.resolveFile("cible.txt");
OutputStream out = flow.getContent().getOutputStream();
IOUtils.copy(is, out);
out.close();
is.close();
file.close();
flow.close();
but i've got an OutOfMemoryError with IOUtils.copy.
Help please!
zizou
--
View this message in context: http://www.nabble.com/OutOfMemoryError-with-VFS-tf2369948.html#a6602245
Sent from the Commons - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by zizou77 <pa...@laposte.net>.
i change my code with:
FileUtil.copyContent(file,flow);
(file size = 1G)
and i've got always OutOfMemoryError.
why?
--
View this message in context: http://www.nabble.com/OutOfMemoryError-with-VFS-tf2369948.html#a6618252
Sent from the Commons - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> i use sftp...
>
Yea, this explains it.
For performance reasons it loaded the whole file into memory.
Please open a jira at http://issues.apache.org/jira so I'll have a look
at it if this is still necessary.
Thanks!
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by zizou77 <pa...@laposte.net>.
i use sftp...
Mario Ivankovits wrote:
>
> Hi!
>> i would like to copy big big file (10G or 20G) with vfs.
>>
> <snip>
>> but i've got an OutOfMemoryError with IOUtils.copy.
>>
> Which filesystems do you use (sftp, http, ftp, .....) ?
>
>
> Ciao,
> Mario
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/OutOfMemoryError-with-VFS-tf2369948.html#a6604172
Sent from the Commons - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> i would like to copy big big file (10G or 20G) with vfs.
>
<snip>
> but i've got an OutOfMemoryError with IOUtils.copy.
>
Which filesystems do you use (sftp, http, ftp, .....) ?
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> My OutOfMemoryError is because it's reading the entire file into memory.
> So i want to modify VFS code but i don't see"stream-based" methodes in
> ChannelSftp class
> Can you help me please?
>
The stream based method for "channel.get(filename, outputStream)" is
simply "inputStream = channel.get(filename)"
The same (just the opposite direction) counts for channel.put()
It looks like there is no stream-based get() for the variant with the
filepointer. Maybe the actual JSCH library has a fix for it already,
else we have to get in touch with the author of this library. But maybe
this didn't affect your current use case.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: OutOfMemoryError with VFS
Posted by zizou77 <pa...@laposte.net>.
I read the source of VFS and in SftpFileObject class there is a TODO in
doGetInputStream() methode:
// TODO - Don't read the entire file into memory. Use the
// stream-based methods on ChannelSftp once they work properly
My OutOfMemoryError is because it's reading the entire file into memory.
So i want to modify VFS code but i don't see"stream-based" methodes in
ChannelSftp class
Can you help me please?
zizou77 wrote:
>
> Hi,
>
> i would like to copy big big file (10G or 20G) with vfs.
> So i use:
>
> FileObject file = flowRepository.resolveFile("src.txt");
> InputStream is = file.getContent().getInputStream();
> FileObject flow = flowRepository.resolveFile("cible.txt");
> OutputStream out = flow.getContent().getOutputStream();
> IOUtils.copy(is, out);
> out.close();
> is.close();
> file.close();
> flow.close();
>
> but i've got an OutOfMemoryError with IOUtils.copy.
>
> Help please!
>
> zizou
>
--
View this message in context: http://www.nabble.com/OutOfMemoryError-with-VFS-tf2369948.html#a6654827
Sent from the Commons - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org