You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/04/30 08:58:03 UTC
[jira] [Created] (HBASE-3834) Store ignores checksum errors when
opening files
Store ignores checksum errors when opening files
------------------------------------------------
Key: HBASE-3834
URL: https://issues.apache.org/jira/browse/HBASE-3834
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.2
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.90.3
If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-3834:
------------------------------------
Assignee: Lars Hofhansl
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: Lars Hofhansl
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448950#comment-13448950 ]
Lars Hofhansl commented on HBASE-3834:
--------------------------------------
Should we close this? There appears to be not much interest in it.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027336#comment-13027336 ]
stack commented on HBASE-3834:
------------------------------
+1
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.3
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3834:
-------------------------
Fix Version/s: 0.92.0
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.92.0, 0.90.5
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457184#comment-13457184 ]
stack commented on HBASE-3834:
------------------------------
Thank you Liang for doing the research.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457176#comment-13457176 ]
Todd Lipcon commented on HBASE-3834:
------------------------------------
Hi Liang. Thanks so much for testing this out. We really appreciate it!
So, let's leave this open to be fixed for an 0.90.x release if we can. Maybe Jon or Ram might be interested in taking it up?
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3834:
-------------------------
Fix Version/s: 0.90.5
Moving into 0.90.5.
This issue is like HBASE-4078.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.5
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448998#comment-13448998 ]
Todd Lipcon commented on HBASE-3834:
------------------------------------
It's still a somewhat scary bug, if it still exists. It causes data to be silently missing from a table. So I hope someone will take interest in it :)
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3834:
-------------------------
Fix Version/s: (was: 0.90.3)
0.90.4
Moving out of 0.90.3. Arm-wrestle Todd if you want to get it back in.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.4
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3834:
--------------------------
Fix Version/s: (was: 0.92.0)
0.92.1
Moving to 0.92.1 release.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.5, 0.92.1
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liang xie updated HBASE-3834:
-----------------------------
Attachment: hbase-3834.tar.gz2
Hi all, i ran a manual test today, it turned out this issue should be gone away in latest version, at least for 0.94.0
Here is my test details:
0)My env: ubuntu 10.10, hbase-0.94.0 release, standalone mode
1)start HBase
2)from hbase shell:
hbase(main):002:0> status
1 servers, 0 dead, 2.0000 average load
hbase(main):003:0> version
0.94.0, r1332822, Tue May 1 21:43:54 UTC 2012
hbase(main):004:0> create 'test','cf'
0 row(s) in 1.1520 seconds
hbase(main):005:0> list 'test'
TABLE
test
1 row(s) in 0.0230 seconds
hbase(main):010:0> put 'test','row1','cf:a','value1'
0 row(s) in 0.0160 seconds
hbase(main):011:0> put 'test','row2','cf:b','value2'
0 row(s) in 0.0070 seconds
hbase(main):012:0> put 'test','row3','cf:c','value3'
0 row(s) in 0.0070 seconds
hbase(main):017:0> scan 'test'
ROW COLUMN+CELL
row1 column=cf:a, timestamp=1347619555652, value=value1
row2 column=cf:b, timestamp=1347619562943, value=value2
row3 column=cf:c, timestamp=1347619576704, value=value3
3 row(s) in 0.0310 seconds
hbase(main):027:0> flush 'test'
0 row(s) in 0.0770 seconds
hbase(main):028:0> exit
3)shutdown hbase,then edit the according files with vim
4)start hbase again
5)from log file, we can see:
2012-09-14 18:54:15,096 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store null
2012-09-14 18:54:15,118 INFO org.apache.hadoop.fs.FSInputChecker: Found checksum error: b[0, 286]=454e010400000012136866696c652e4156475f56414c55455f4c454e0104000000040d6866696c652e4c4153544b455901130004726f77330263666300000139c462db8004a887830f545241424c4b2224000000000000010f00000000000000a600000001000000000000001f0000000000000000000002f600000000000000040000000200000001000000000000000000000000000000006f72672e6170616368652e6861646f6f702e68626173652e4b657956616c7565244b6579436f6d70617261746f7200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000020a
org.apache.hadoop.fs.ChecksumException: Checksum error: file:/tmp/hbase/test/4c9f2cfda63b2e9785815ed2e841d052/cf/269b59832d68465687ebce880026a301 at 512
at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
...
2012-09-14 18:54:15,123 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'test,,1347619334483.4c9f2cfda63b2e9785815ed2e841d052.', STARTKEY => '', ENDKEY => '', ENCODED => 4c9f2cfda63b2e9785815ed2e841d052,} failed, marking as FAILED_OPEN in ZK
2012-09-14 18:54:15,123 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:48394-0x139c46a10d40001 Attempting to transition node 4c9f2cfda63b2e9785815ed2e841d052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
2012-09-14 18:54:15,137 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:48394-0x139c46a10d40001 Successfully transitioned node 4c9f2cfda63b2e9785815ed2e841d052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
2012-09-14 18:54:15,138 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=xieliang,48394,1347620050131, region=4c9f2cfda63b2e9785815ed2e841d052
2012-09-14 18:54:15,141 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 4c9f2cfda63b2e9785815ed2e841d052
2012-09-14 18:54:15,141 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=test,,1347619334483.4c9f2cfda63b2e9785815ed2e841d052. state=CLOSED, ts=1347620055124, server=xieliang,48394,1347620050131
2012-09-14 18:54:15,141 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:41380-0x139c46a10d40000 Creating (or updating) unassigned node for 4c9f2cfda63b2e9785815ed2e841d052 with OFFLINE state
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl reassigned HBASE-3834:
------------------------------------
Assignee: liang xie (was: Lars Hofhansl)
Assigned to Liang by request :)
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-3834:
------------------------------------------
Moving to 0.90.7
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.6
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456262#comment-13456262 ]
liang xie commented on HBASE-3834:
----------------------------------
If nobody have a chance to try, i'll verify it at 0.90.6 in next monday
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457650#comment-13457650 ]
Hadoop QA commented on HBASE-3834:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545531/HBASE-3834.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2888//console
This message is automatically generated.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: HBASE-3834.patch, hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-3834:
----------------------------------
Fix Version/s: (was: 0.90.7)
0.90.8
No activity, moving to 0.90.8
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449432#comment-13449432 ]
Lars Hofhansl commented on HBASE-3834:
--------------------------------------
Sorry... Assigned it to me first (by accident), then to Liang.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449391#comment-13449391 ]
liang xie commented on HBASE-3834:
----------------------------------
Hi Todd, could you assign it to me please? it sounds interesting to me somehow:)
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456000#comment-13456000 ]
Todd Lipcon commented on HBASE-3834:
------------------------------------
Great to see this is fixed in 0.94.
Does someone have time to try this on the latest 0.90.x, which is still in production a lot of places?
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028859#comment-13028859 ]
stack commented on HBASE-3834:
------------------------------
I think we should punt on this one. Its rare.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.3
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028630#comment-13028630 ]
Todd Lipcon commented on HBASE-3834:
------------------------------------
Sorry, not likely I have time to do this (HA taking up my time these days). I don't have the logs handy, but it's easy to produce:
- Start HBase standalone with default configs (ie rootdir file:///tmp/hbase-foo)
- Insert some data, flush
- Shutdown hbase
- vim /tmp/hbase-foo/path/to/some/hfile. Randomly replace a few characters here and there
- Start hbase
- Data gone :(
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.3
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028536#comment-13028536 ]
Jean-Daniel Cryans commented on HBASE-3834:
-------------------------------------------
Do you have time to work on this for 0.90.3 Todd? If not, would it be possible to see the error message that's being logged and the stack trace?
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.3
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456750#comment-13456750 ]
liang xie commented on HBASE-3834:
----------------------------------
Hi Todd, i tried on 0.90.5&0.90.6, the data was lost silently. But on 0.92.0, the result was just same as the above 0.94.0, so this issue seems exists on 0.90.x(or older) serials only.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3834:
-------------------------
Fix Version/s: (was: 0.90.4)
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-3834:
------------------------------------------
Fix Version/s: (was: 0.90.6)
0.90.7
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.7
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-3834:
----------------------------------
No activity, moving to 0.90.8
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449406#comment-13449406 ]
stack commented on HBASE-3834:
------------------------------
Lars, you assigned it back to yourself again. Was that intentional? I thought you were trying to pass it to Liang?
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064343#comment-13064343 ]
stack commented on HBASE-3834:
------------------------------
Moving out of 0.90.4. Arm-wrestle me Todd if you want to get it back in (Arm-wrestle threat worked for moving it out of 0.90.3).
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.90.4
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455731#comment-13455731 ]
liang xie commented on HBASE-3834:
----------------------------------
IMHO, this is up to Todd's initial expectation.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liang xie updated HBASE-3834:
-----------------------------
Attachment: HBASE-3834.patch
Hi Todd&Stack, actually, this issue was caused by the CATCH statement in loadStoreFiles().
After applied the attached patch, i could observe just the same behivor as 0.92&0.94. and i also ensure there's no CATCH in 0.92&0.94's loadStoreFiles().
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: HBASE-3834.patch, hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "liang xie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liang xie updated HBASE-3834:
-----------------------------
Affects Version/s: 0.90.3
0.90.4
0.90.5
0.90.6
Status: Patch Available (was: Open)
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.6, 0.90.5, 0.90.4, 0.90.3, 0.90.2
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: HBASE-3834.patch, hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3834) Store ignores checksum errors when
opening files
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3834:
-------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.90 branch. Thanks for digging in Liang.
> Store ignores checksum errors when opening files
> ------------------------------------------------
>
> Key: HBASE-3834
> URL: https://issues.apache.org/jira/browse/HBASE-3834
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.90.5, 0.90.6
> Reporter: Todd Lipcon
> Assignee: liang xie
> Priority: Critical
> Fix For: 0.90.8
>
> Attachments: HBASE-3834.patch, hbase-3834.tar.gz2
>
>
> If you corrupt one of the storefiles in a region (eg using vim to muck up some bytes), the region will still open, but that storefile will just be ignored with a log message. We should probably not do this in general - better to keep that region unassigned and force an admin to make a decision to remove the bad storefile.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira