You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ning Li (JIRA)" <ji...@apache.org> on 2008/09/25 02:05:44 UTC
[jira] Created: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Bug in FSInputChecker makes it possible to read from an invalid buffer
----------------------------------------------------------------------
Key: HADOOP-4271
URL: https://issues.apache.org/jira/browse/HADOOP-4271
Project: Hadoop Core
Issue Type: Bug
Components: fs
Reporter: Ning Li
Priority: Minor
Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: hadoop-4271-branch-18.patch
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-4271:
----------------------------------
Fix Version/s: (was: 0.20.0)
(was: 0.18.2)
(was: 0.19.0)
0.17.3
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: (was: HADOOP-4277-branch-18.patch)
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: HADOOP-4277-branch-18.patch
0.17 and 0.18 versions of Ning's patch are attached.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634526#action_12634526 ]
Ning Li commented on HADOOP-4271:
---------------------------------
> > You really don't need change ChecksumFileSystem at all. You don't need to test with verifyChecksum off I think. It will make the patch much simpler.
> Before the patch, the default of verifyChecksum in FSInputChecker is false. I made the change in order to test with verifyChecksum being true.
Should the default of verifyChecksum be true? If so and if we only test with verifyChecksum being true, then yes, the patch can be much simpler.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: hadoop-4271-branch-17.patch
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636032#action_12636032 ]
Hudson commented on HADOOP-4271:
--------------------------------
Integrated in Hadoop-trunk #620 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/])
fix jira number for
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Comment: was deleted
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Li updated HADOOP-4271:
----------------------------
Attachment: hadoop-4271.patch
The updated patch with HADOOP-4277 committed.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634390#action_12634390 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
Good catch. This can happen when {{seek()}} is used. A few things:
- I think (more) correct fix is to reset 'pos' and 'count' _before_ calling {{ readChecksumChunk()}}.. since once it is called, buffer is no longer valid (say there is some exception and user seeks again).
- You can generate 1500 bytes of random data in the test itself rather than requiring two extra files to be committed.
- Currently 'runTest()' fails only with checksum disabled. Same bug exists with checksum enabled as well, which is the common case. Can you change the test so that it fails with checksum enabled as well?
- There is already a {{TestFSInputChecksum.java}} in hdfs. Can you add this case there?
- Minor : using IOUtils.readFully() will reduce the code for the test.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Ning Li
> Priority: Minor
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634566#action_12634566 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
> I'm confused. Do you want me to fix HADOOP-4277?
:). I will fix HADOOP-4277. The patch there will conflict with the patch here. Please provide an updated patch once HADOOP-4277 is committed (today or tomorrow).
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634557#action_12634557 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
Ning,
I filed HADOOP-4277 for 'verifyChecksum'. Could you assign it to yourself if you don't mind fixing it? I will mark this jira dependent on it.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Li updated HADOOP-4271:
----------------------------
Attachment: .data_testFSInputChecker.crc
data_testFSInputChecker
hadoop-4271.patch
Attached is the patch for this issue. A test is added to verify the fix. Two data files used by the test is also included. All tests pass with the patch.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Ning Li
> Priority: Minor
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: (was: HADOOP-4277-branch-17.patch)
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635538#action_12635538 ]
Hairong Kuang commented on HADOOP-4271:
---------------------------------------
> The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
I do not understand why this is a bug. Only when the buffer does not have any available byte, data are read to the user buffer directly. So there is no need to invalidate the buffer.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635780#action_12635780 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
Just a clarification that this does not affect HDFS reads since DFSInputStream has its own seek().
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635001#action_12635001 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
Ning, HADOOP-4277 is committed. Would you be able to update this patch today? You can create a patch for trunk and I will make patches for 0.17 and 0.18.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634528#action_12634528 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
> Before the patch, the default of verifyChecksum in FSInputChecker is false.
This is a surprise. It should be true by default. You might have found another serious bug! Let me confirm.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635044#action_12635044 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
test-patch output : {noformat}
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{noformat}
will commit the patch today.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: (was: HADOOP-4277-branch-18.patch)
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634521#action_12634521 ]
Ning Li commented on HADOOP-4271:
---------------------------------
> You really don't need change ChecksumFileSystem at all. You don't need to test with verifyChecksum off I think. It will make the patch much simpler.
Before the patch, the default of verifyChecksum in FSInputChecker is false. I made the change in order to test with verifyChecksum being true.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: HADOOP-4277-branch-18.patch
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: hadoop-4271.patch
Resubmitting Ning's patch trunk so that Hudson picks the right one.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634562#action_12634562 ]
Ning Li commented on HADOOP-4271:
---------------------------------
I'm confused. :) Do you want me to fix HADOOP-4277?
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
+1.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635541#action_12635541 ]
Hairong Kuang commented on HADOOP-4271:
---------------------------------------
Ok I see that it effects "seek". This makes sense.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-4271:
------------------------------------
Assignee: Ning Li
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Priority: Blocker (was: Critical)
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: HADOOP-4277-branch-18-correction.patch
The patch attached for 0.18 above is incomplete. I committed attached HADOOP-4277-branch-18-correction.patch in addition.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-18-correction.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: hadoop-4271-branch-18.patch
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Affects Version/s: 0.16.0
Fix Version/s: 0.20.0
0.19.0
0.18.2
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Critical
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634515#action_12634515 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
Thanks Ning.
> The price is that I have to make ChecksumFileSystem.open(Path f, int bufferSize, boolean verifyChecksum) a public method.
You really don't need change ChecksumFileSystem at all. You don't need to test with verifyChecksum off I think. It will make the patch much simpler.
Minor: for the same reason, it might be better to set count to zero in fill() as well. We could remove resetting these inside {{readChecksumChunk()}} (in case of checksum exception).
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Attachment: HADOOP-4277-branch-17.patch
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, HADOOP-4277-branch-17.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634558#action_12634558 ]
Raghu Angadi commented on HADOOP-4271:
--------------------------------------
I assigned HADOOP-4277 to myself. Thanks Ning.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634483#action_12634483 ]
Ning Li commented on HADOOP-4271:
---------------------------------
I will address the comments in the next patch.
> There is already a TestFSInputChecksum.java in hdfs. Can you add this case there?
I see a TestFSInputChecker.java in hdfs. I'll merge the two. But should I move it to fs? Since everything tested is in fs?
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Priority: Critical (was: Minor)
Marking this critical since wrong data returned is a serious bug.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Reporter: Ning Li
> Priority: Critical
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4271:
---------------------------------
Resolution: Fixed
Release Note: Checksum input stream can sometimes return invalid data to the user.
Status: Resolved (was: Patch Available)
I just committed this. Thanks Ning!
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Li updated HADOOP-4271:
----------------------------
Attachment: hadoop-4271.patch
This patch addresses all the comments. I leave TestFSInputChecker.java in hdfs since it tests on MiniDFSCluster as well. The price is that I have to make ChecksumFileSystem.open(Path f, int bufferSize, boolean verifyChecksum) a public method.
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0, 0.20.0
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4271) Bug in FSInputChecker makes it
possible to read from an invalid buffer
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-4271:
--------------------------------
Fix Version/s: 0.18.2
> Bug in FSInputChecker makes it possible to read from an invalid buffer
> ----------------------------------------------------------------------
>
> Key: HADOOP-4271
> URL: https://issues.apache.org/jira/browse/HADOOP-4271
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Blocker
> Fix For: 0.17.3, 0.18.2
>
> Attachments: .data_testFSInputChecker.crc, data_testFSInputChecker, hadoop-4271-branch-17.patch, hadoop-4271-branch-18.patch, hadoop-4271-branch-18.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch, hadoop-4271.patch
>
>
> Bug in FSInputChecker makes it possible to read from an invalid buffer. The buffer in FSInputChecker becomes invalid when readChecksumChunk is used to read a chunk to a user buffer directly. Currently, it's not marked as invalid in this case and may be read subsequently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.