[jira] [Updated] (MAPREDUCE-5075) DistCp leaks input file handles

["Chris Nauroth (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Nauroth updated MAPREDUCE-5075:
-------------------------------------

    Attachment: MAPREDUCE-5075.1.patch

Here is a patch with the following changes:

# {{RetriableFileCopyCommand}} - This is just code clean-up.  The {{copyBytes}} private method accepted a flag as an argument to control whether or not to close the streams after copying.  This method was only ever called from {{copyToTmpFile}} with a hard-coded true.  I removed the flag from the method signature and changed the code so that it closes the streams unconditionally.
# {{ThrottledInputStream}} - Override {{close}} so that it closes the wrapped stream.
# {{TestIntegration}} - This code was not creating the target file correctly.  {{target}} contains a fully qualified path.  Inside {{createFiles}}, it prepends the test root again.  This would be 2 fully qualified paths appended to each other.  On Windows, the result would look like C:\project\target\C:\project\target.  The second ':' makes the filename invalid.

With this patch, all DistCp tests pass consistently on Mac and Windows.

                
> DistCp leaks input file handles
> -------------------------------
>
>                 Key: MAPREDUCE-5075
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5075
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: MAPREDUCE-5075.1.patch
>
>
> DistCp wraps the {{InputStream}} for each input file it reads in an instance of {{ThrottledInputStream}}.  This class does not close the wrapped {{InputStream}}.  {{RetriableFileCopyCommand}} guarantees that the {{ThrottledInputStream}} gets closed, but without closing the underlying wrapped stream, it still leaks a file handle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira