You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "David Sitsky (JIRA)" <ji...@apache.org> on 2008/08/04 08:27:44 UTC
[jira] Created: (IO-175) IOUtils.doCopyFile() issues with very
large files and closing file input streams
IOUtils.doCopyFile() issues with very large files and closing file input streams
--------------------------------------------------------------------------------
Key: IO-175
URL: https://issues.apache.org/jira/browse/IO-175
Project: Commons IO
Issue Type: Bug
Components: Utilities
Reporter: David Sitsky
Priority: Minor
Fix For: 2.x
I've noticed the code in http://svn.apache.org/repos/asf/commons/proper/io/trunk/src/java/org/apache/commons/io/FileUtils.java for doCopyFile() contains:
output.transferFrom(input, 0, input.size());
I know from experience (I had code that did this previously) this will not work on large files, at least under Windows. By default, the transferFrom() method will try to create a single memory map equal to the size of the input. If you are running a 32-bit process, and are trying to copy a file gigabytes in size, this will fail as you won't have enough virtual address space to create the memory map. We had to use transferFrom() in smaller chunk sizes to work around this.
Also I believe in the code of this method, the file input streams need to be closed explicitly, since closing the channel objects that are derived from them will not close any resources held by the file input stream objects themselves.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (IO-175) IOUtils.doCopyFile() issues with very
large files and closing file input streams
Posted by "Niall Pemberton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IO-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Niall Pemberton resolved IO-175.
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 2.x)
2.0
Assignee: Niall Pemberton
Thanks for pointing this out, I have limited the transfer to 50MB chunks and explicitly closed the streams now:
http://svn.apache.org/viewvc?view=rev&revision=723199
> IOUtils.doCopyFile() issues with very large files and closing file input streams
> --------------------------------------------------------------------------------
>
> Key: IO-175
> URL: https://issues.apache.org/jira/browse/IO-175
> Project: Commons IO
> Issue Type: Bug
> Components: Utilities
> Reporter: David Sitsky
> Assignee: Niall Pemberton
> Priority: Minor
> Fix For: 2.0
>
>
> I've noticed the code in http://svn.apache.org/repos/asf/commons/proper/io/trunk/src/java/org/apache/commons/io/FileUtils.java for doCopyFile() contains:
> output.transferFrom(input, 0, input.size());
> I know from experience (I had code that did this previously) this will not work on large files, at least under Windows. By default, the transferFrom() method will try to create a single memory map equal to the size of the input. If you are running a 32-bit process, and are trying to copy a file gigabytes in size, this will fail as you won't have enough virtual address space to create the memory map. We had to use transferFrom() in smaller chunk sizes to work around this.
> Also I believe in the code of this method, the file input streams need to be closed explicitly, since closing the channel objects that are derived from them will not close any resources held by the file input stream objects themselves.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.