You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Michael Tamm (JIRA)" <ji...@apache.org> on 2009/03/12 10:30:51 UTC
[jira] Created: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
-----------------------------------------------------------------------------------------------------
Key: HADOOP-5476
URL: https://issues.apache.org/jira/browse/HADOOP-5476
Project: Hadoop Core
Issue Type: Bug
Components: io
Affects Versions: 0.19.1
Reporter: Michael Tamm
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Tamm updated HADOOP-5476:
---------------------------------
Attachment: updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5476) calling new
SequenceFile.Reader(...) leaves an InputStream open, if the given sequence
file is broken
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706667#action_12706667 ]
Chris Douglas commented on HADOOP-5476:
---------------------------------------
If seek or init throws a RuntimeException, it's wrong. Unfortunately, init _does_ throw IllegalArgumentException if the codec lookup fails. *sigh*
You're right, try-finally is required. I'd still use IOUtils::cleanup to close the stream and drop the comments, though.
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5476:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Michael
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Fix For: 0.21.0
>
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Tamm updated HADOOP-5476:
---------------------------------
Affects Version/s: 0.20.0
Status: Patch Available (was: Open)
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.0, 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5476) calling new
SequenceFile.Reader(...) leaves an InputStream open, if the given sequence
file is broken
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707121#action_12707121 ]
Chris Douglas commented on HADOOP-5476:
---------------------------------------
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
Running the unit tests...
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5476) calling new
SequenceFile.Reader(...) leaves an InputStream open, if the given sequence
file is broken
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707494#action_12707494 ]
Hudson commented on HADOOP-5476:
--------------------------------
Integrated in Hadoop-trunk #830 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/])
. Close the underlying InputStream in SequenceFile::Reader when
the constructor throws an exception. Contributed by Michael Tamm
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Fix For: 0.21.0
>
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Tamm updated HADOOP-5476:
---------------------------------
Status: Patch Available (was: Open)
I attached the third version of the patch, which uses IOUtils.cleanup now.
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.0, 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476_which_uses_IOUtils_cleanup.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5476) calling new
SequenceFile.Reader(...) leaves an InputStream open, if the given sequence
file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706618#action_12706618 ]
Michael Tamm commented on HADOOP-5476:
--------------------------------------
Hmm, that would not cover all error cases:
if a RuntimeException is thrown by seek(start) or init(tempReader),
in would not be closed.
For me it is a best practice to use try-finally to close opened resources.
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1, 0.20.0
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5476:
----------------------------------
Status: Open (was: Patch Available)
Thanks for updating the patch; I meant the first try/finally, which is actually distinguishing the case where an IOException is thrown from the case where it is not. That is better expressed as:
{code}
this.conf = conf;
try {
seek(start);
this.end = in.getPos() + length;
init(tempReader);
} catch (IOException e) {
try {
in.close();
} catch (IOException ce) {
LOG.info("Exception from close in init: ", ce);
}
throw e;
}
{code}
or even more succinctly:
{code}
try {
seek(start);
this.end = in.getPos() + length;
init(tempReader);
} catch (IOException e) {
IOUtils.cleanup(LOG, in);
throw e;
}
{code}
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.0, 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-5476:
---------------------------------
Assignee: Michael Tamm
Status: Patch Available (was: Open)
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Tamm updated HADOOP-5476:
---------------------------------
Attachment: updated_test_and_fix_for_HADOOP-5476.patch
The updated patch addresses all mentioned problems and the tests no longer throws a NPE.
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch, updated_test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Michael Tamm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Tamm updated HADOOP-5476:
---------------------------------
Attachment: test_and_fix_for_HADOOP-5476.patch
I have attached a patch, which
* adds the new method testCloseForErroneousSequenceFile to TestSequenceFile.java to check this bug
* fixes the SequenceFile.Reader constructor
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1
> Reporter: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5476) calling new SequenceFile.Reader(...)
leaves an InputStream open, if the given sequence file is broken
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5476:
----------------------------------
Status: Open (was: Patch Available)
Sorry for the long delay in reviewing this.
* TestSequenceFile fails on trunk:
{noformat}
Testcase: testCloseForErroneousSequenceFile took 0.102 sec
Caused an ERROR
null
java.lang.NullPointerException
at org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.read(RawLocalFileSystem.java:91)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:142)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:134)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1453)
at org.apache.hadoop.io.TestSequenceFile$1.openFile(TestSequenceFile.java:494)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1430)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1423)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1418)
at org.apache.hadoop.io.TestSequenceFile$1.<init>(TestSequenceFile.java:491)
at org.apache.hadoop.io.TestSequenceFile.testCloseForErroneousSequenceFile(TestSequenceFile.java:491)
{noformat}
If the unit test obtains the LFS by calling {{FileSystem.getLocal(conf)}} then everything works as designed.
* The conf assignment can be outside the try block
* Instead of try/finally, catching IOException, attempting a close, logging- rather than ignoring- exceptions thrown from close, then re-throwing the IOException would be preferred.
> calling new SequenceFile.Reader(...) leaves an InputStream open, if the given sequence file is broken
> -----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5476
> URL: https://issues.apache.org/jira/browse/HADOOP-5476
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.1
> Reporter: Michael Tamm
> Assignee: Michael Tamm
> Attachments: test_and_fix_for_HADOOP-5476.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.