You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (JIRA)" <ji...@apache.org> on 2012/07/17 14:36:34 UTC
[jira] [Created] (HBASE-6401) HBase may lose edits after a crash if
used with HDFS 1.0.3 or older
nkeywal created HBASE-6401:
------------------------------
Summary: HBase may lose edits after a crash if used with HDFS 1.0.3 or older
Key: HBASE-6401
URL: https://issues.apache.org/jira/browse/HBASE-6401
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Environment: all
Reporter: nkeywal
Priority: Critical
This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
- So we have in hdfs one (dead) process writing with another process reading.
- But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
- So we may miss all the writes to the last block if we try to connect to the dead DN.
hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
hdfs branch-2 or trunk: we should not have the issue (but not tested)
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419579#comment-13419579 ]
stack commented on HBASE-6401:
------------------------------
@Nkeywal We need another patch on top of hdfs-3222?
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419787#comment-13419787 ]
stack commented on HBASE-6401:
------------------------------
@Nkeywal Sounds great. A patch/JIRA will give us something to rally around.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474234#comment-13474234 ]
nkeywal commented on HBASE-6401:
--------------------------------
Branch 1.1 contains the fix for this issue. So I would tend to think that we can resolved this at won't fix as this jira mentions the branch 1.0.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal reassigned HBASE-6401:
------------------------------
Assignee: nkeywal
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419356#comment-13419356 ]
nkeywal commented on HBASE-6401:
--------------------------------
@stack
bq. Does svn blame/git bisecting not turn up the issue that fixed this?
Will try.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463144#comment-13463144 ]
nkeywal commented on HBASE-6401:
--------------------------------
HDFS-3701 has just been fixed, so we may have a reasonable hdfs 1.1 version as HDFS-3703 made it as well. We need HDFS-3912 to be complete from a failure management point of view. Then there is the question of durability...
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419423#comment-13419423 ]
nkeywal commented on HBASE-6401:
--------------------------------
HDFS-3222 is not exactly this one, but not far, and fixed on 2.0 as well.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416223#comment-13416223 ]
Zhihong Ted Yu commented on HBASE-6401:
---------------------------------------
The test is for hadoop.
Is there HADOOP- JIRA for this bug ?
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419398#comment-13419398 ]
nkeywal commented on HBASE-6401:
--------------------------------
@stack The oldest version of DFSInputStream.java (it"s split from DFSClient) is one year old and seems ok on trunk.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419358#comment-13419358 ]
nkeywal commented on HBASE-6401:
--------------------------------
HBASE-6435 will lower the probability to get the issue but will not solve it totally.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416241#comment-13416241 ]
nkeywal commented on HBASE-6401:
--------------------------------
Yeah, I wrote it for hadoop, but then saw it was fixed on their trunk, so I didn't created it. But I haven't found a related jira. We could want to fix it on 1.0.3, adding the ordering I was mentionning on the dev list (put the DN on the same box as the RS as the last locations to make sure we don't use a dead DN).
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6401) HBase may lose edits after a crash if
used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-6401:
---------------------------
Attachment: TestReadAppendWithDeadDN.java
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463115#comment-13463115 ]
Lars Hofhansl commented on HBASE-6401:
--------------------------------------
Hadoop-2 has other issues, though (see last few comments on HDFS-744).
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417149#comment-13417149 ]
Lars Hofhansl commented on HBASE-6401:
--------------------------------------
This needs to be fixed in a hdfs. If it is fixed in Hadoop-2 there should be a jira to backport the change.
When I did HDFS-744 I found Hadoop-1 and Hadoop-2 quite different in the way it handled packet shipping from the DFSClient, so the work might be non-trivial.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419585#comment-13419585 ]
nkeywal commented on HBASE-6401:
--------------------------------
On 1.x, yes, I think that backporting hdfs-3222 won't be enough. On 2.0, it seems it's ok, even if I can't find the good soul who fixed it. As I can't find a jira, I can create a new one & propose a fix specific to branch 1.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal resolved HBASE-6401.
----------------------------
Resolution: Won't Fix
HDFS 1.1.x includes the fix.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6401) HBase may lose edits after a crash
if used with HDFS 1.0.3 or older
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417161#comment-13417161 ]
stack commented on HBASE-6401:
------------------------------
bq. I haven't found the hdfs jira for this.
Does svn blame/git bisecting not turn up the issue that fixed this?
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>
> Key: HBASE-6401
> URL: https://issues.apache.org/jira/browse/HBASE-6401
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.0
> Environment: all
> Reporter: nkeywal
> Priority: Critical
> Attachments: TestReadAppendWithDeadDN.java
>
>
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira