You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2012/05/30 22:14:23 UTC
[jira] [Created] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Colin Patrick McCabe created HDFS-3479:
------------------------------------------
Summary: backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
Key: HDFS-3479
URL: https://issues.apache.org/jira/browse/HDFS-3479
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3479:
------------------------------
Fix Version/s: 1.2.0
Issue Type: Improvement (was: Bug)
Hadoop Flags: Reviewed
I've committed this to branch-1. Thanks Colin.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.2.0
>
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins resolved HDFS-3479.
-------------------------------
Resolution: Fixed
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.2.0
>
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287054#comment-13287054 ]
Hadoop QA commented on HDFS-3479:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530479/HDFS-3479-b1.002.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2561//console
This message is automatically generated.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287739#comment-13287739 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
Eek, looks like you're right. From DataInputStream#read(byte b[], int off, int len) throws IOException:
{code}
* Reads up to <code>len</code> bytes of data from the contained
* input stream into an array of bytes. An attempt is made to read
* as many as <code>len</code> bytes, but a smaller number may be read,
* possibly zero. The number of bytes actually read is returned as an
* integer.
{code}
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286080#comment-13286080 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
Hi Nicholas,
In the patch it says "We don't check the last two megabytes of the edit log, in case the NameNode crashed while writing to the edit log."
Basically, if we crash while writing to the end of the log, the underlying filesystem does not give us the guarantees we would need to get every byte perfect. Consider the following sequence of events:
1. NN allocates an extra 1 MB at the end of the file and fills it with 0xff bytes
2. NN writes an opcode to the edit log file. It happens to span two sectors on the hard disk
3. The kernel writes the second half of the opcode to disk
4. system crash
In this case, we're left with a file that looks like this:
{code}
0xff 0xff 0xff 0xff ... [opcode bytes]... 0xff 0xff 0xff
{code}
This would clearly fail validation. Hence the NameNode would fail to start, even though no data has been lost (the opcode was never acked to the client). This would be a serious problem. UNCHECKED_REGION_LENGTH fixes this problem.
We can't control the order in which the kernel flushes sectors out of the buffer cache and on to the hard disk. We can set up barriers (that is what fsync is), but control of the ordering is beyond us.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287805#comment-13287805 ]
Hadoop QA commented on HDFS-3479:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530615/HDFS-3479-b1.003.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2574//console
This message is automatically generated.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290416#comment-13290416 ]
Todd Lipcon commented on HDFS-3479:
-----------------------------------
Patch looks good to me. +1. Any further comments, Nicholas? If not, I'll commit later today.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292158#comment-13292158 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
bq. One more question: why the trunk patch do not have the UNCHECKED_REGION_LENGTH stuff?
trunk handles this differently, using recovery mode to ensure that we don't stop reading the edit log early. This is possible because recovery mode is a lot more advanced in trunk than in branch-1.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.2.0
>
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290341#comment-13290341 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
I ran all the tests, and also test-patch. There were no issues. We should be ready to commit, thanks everyone.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292147#comment-13292147 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
One more question: why the trunk patch do not have the UNCHECKED_REGION_LENGTH stuff?
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.2.0
>
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290558#comment-13290558 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
bq. written could be less than fill.capacity(). Is it a problem?
I have an upcoming change, HDFS-3510, which fixes this. I think it makes sense to submit this patch as-is, and work on the FileChannel#write issues separately. It should make reviews easier and quicker than combining the changes. Makes sense?
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292148#comment-13292148 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
BTW, I have created HDFS-3521 for further improvement.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.2.0
>
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287040#comment-13287040 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
That's a good point. It should be 1MB, since we're never writing more than that to disk in any single call to FSEditLog#flushAndSync in branch-1.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3479:
---------------------------------------
Attachment: HDFS-3479-b1.003.patch
* verifyEndOfLog: handle short reads
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290439#comment-13290439 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
{code}
+ int written = fc.write(fill, position);
{code}
written could be less than fill.capacity(). Is it a problem?
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286250#comment-13286250 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
Why UNCHECKED_REGION_LENGTH is 2MB but not the preallocation length (1MB)?
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3479:
---------------------------------------
Attachment: HDFS-3335-b1.005.patch
* fix the way we do pre-allocation (formerly, we were creating sparse files in branch-1, not actually reserving the space we thought we were.)
* an OP_INVALID is now an error unless:
1. it appears in the last 2 MB of the file
or
2. it is followed only by 0x0 and 0xff bytes, until the end of the file.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3479:
---------------------------------------
Status: Patch Available (was: Open)
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289787#comment-13289787 ]
Colin Patrick McCabe commented on HDFS-3479:
--------------------------------------------
ran tests TestCheckpoint \
TestEditLog \
TestNameNodeRecovery \
TestEditLogLoading \
TestNameNodeMXBean \
TestSaveNamespace \
TestSecurityTokenEditLog \
TestStorageDirectoryFailure \
TestStorageRestore
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286051#comment-13286051 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
Why there is a UNCHECKED_REGION_LENGTH?
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290676#comment-13290676 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
I am okay if you are going to fix it in HDFS-3510. Thanks.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289849#comment-13289849 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3479:
----------------------------------------------
Please run all the tests and also test-patch. That are what Jenkins run.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-3479:
---------------------------------------
Attachment: HDFS-3479-b1.002.patch
* reduce the unchecked region to 1MB.
* add PREALLOCATION_LENGTH constant rather than hard-coding
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3479) backport HDFS-3335 (check for edit log
corruption at the end of the log) to branch-1
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3479:
------------------------------
Status: Open (was: Patch Available)
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch, HDFS-3479-b1.003.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286009#comment-13286009 ]
Hadoop QA commented on HDFS-3479:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530264/HDFS-3335-b1.005.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2548//console
This message is automatically generated.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3479) backport HDFS-3335 (check for edit
log corruption at the end of the log) to branch-1
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287703#comment-13287703 ]
Todd Lipcon commented on HDFS-3479:
-----------------------------------
{code}
+ int numRead = in.read(buf, 0, (int)amt);
+ if (numRead < amt) {
{code}
{{read}} isn't guaranteed to return all bytes requested, even if there are more bytes in the file. So I don't think this check is necessary -- instead just check numRead >= 0, I think.
Otherwise looks good to me.
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3479
> URL: https://issues.apache.org/jira/browse/HDFS-3479
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3335-b1.005.patch, HDFS-3479-b1.002.patch
>
>
> backport HDFS-3335 (check for edit log corruption at the end of the log) to branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira