You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Hsieh (JIRA)" <ji...@apache.org> on 2012/08/15 00:04:38 UTC
[jira] [Created] (HBASE-6586) Quarantine Corrupted HFiles
Jonathan Hsieh created HBASE-6586:
-------------------------------------
Summary: Quarantine Corrupted HFiles
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
We've encountered a few upgrades that get stuck from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
trying again
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443085#comment-13443085 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #152 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/152/])
HBASE-6586 Quarantine Corrupted HFiles with hbck (Revision 1377941)
Result = FAILURE
jmhsieh :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: hbase-6586-94-v8.patch)
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443350#comment-13443350 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-0.92-security #118 (See [https://builds.apache.org/job/HBase-0.92-security/118/])
HBASE-6586 Quarantine Corrupted HFiles with hbck (Revision 1378149)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-92-v8a.patch
Previous 92 version was missing new HBaseIOException.java file.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436367#comment-13436367 ]
Jimmy Xiang commented on HBASE-6586:
------------------------------------
Jon posted it on review board: https://reviews.apache.org/r/6664/
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: hbase-6586.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434693#comment-13434693 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
I basically agree with the dangerous default argument -- which is why I'm suggesting a mode for it. (Similar to upgrade mode in a hdfs nn).
Another alternate with what Andy suggested -- making it part of some hbck mode where all hfiles are checked. Here is it is definitely admin initiated.
Another suggestion was to recover data from truncated HFiles -- which is something to consider (but likely won't come until we have an directed need for it).
WRT HDFS-3731 - I've done a few combos of running job that loads data, kill hbase/hdfs in safe and unsafe ways, and then upgrade but still haven't been able to duplicate same HFile error. In these scenarios, I've had both log files and hfiles with block-being-written state problems. I believe Jimmy has dealt with the hlog problems with a dist log splitting fix.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442807#comment-13442807 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
On build 2703, two tests hung:
{code}
./findHangingTest.sh https://builds.apache.org/job/PreCommit-HBASE-Build/2703/console
Hanging test: Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Hanging test: Running org.apache.hadoop.hbase.rest.TestScannersWithFilters
{code}
On build 2705, the test passes for me locally.
I believe these are flakey.
I've run them all locally on 94/trunk and they all basically pass. The 92 version sees to semi-frequently hangs/bails on TestMergeTool. This has been taking >5 minutes in 94, and >8 mins in 92). I don't think these are related.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: 0001-hbase-6568-hbck-quarantine-v6.patch
v2, adds checks/warning for deleted files.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448351#comment-13448351 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/])
HBASE-6586 Quarantine Corrputed HFiles with hbck (Revision 1377942)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Affects Version/s: 0.92.1
Fix Version/s: 0.94.3
0.96.0
0.92.2
Hadoop Flags: Reviewed
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442936#comment-13442936 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
0.92 version passes (TestSplitTranscationOnCluster flaky but passes on separate local run).
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Benoit Sigoure (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440930#comment-13440930 ]
Benoit Sigoure commented on HBASE-6586:
---------------------------------------
Not sure why {{HBaseIOException}} would be added in a JIRA about "Quarantine Corrupted HFiles", but yes that would be nice to have.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436120#comment-13436120 ]
Andrew Purtell commented on HBASE-6586:
---------------------------------------
bq. Ideally when we have a none-recoverable failure like corrupted hfiles, we'd transition to QUARANTINED instead of FAILED_OPEN and stay there until an admin fixes the problem.
+1 on this too.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-92-v8b.patch
Cleaned up some a merge conflict that was in a comment that I had missed.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442916#comment-13442916 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-0.94 #441 (See [https://builds.apache.org/job/HBase-0.94/441/])
HBASE-6586 Quarantine Corrputed HFiles with hbck (Revision 1377942)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436081#comment-13436081 ]
Andrew Purtell commented on HBASE-6586:
---------------------------------------
[~jmhsieh] Do you think it reasonable and worth it to add a HFile corruption detect-and-quarantine flag and mode to hbck?
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436118#comment-13436118 ]
Andrew Purtell commented on HBASE-6586:
---------------------------------------
bq. I do, I built it yesterday and am polishing up the test currently, and looking for more corner cases. It is a separate phase (it ignores metadata) and is kind of analogous to "checking every block" on a file system
Sounds great! Thanks Jon.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v4.patch
v4, changes io exception to illegal argument exception.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v8.patch
Previous run didn't pass but when tested locally it passed. Trying again.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443274#comment-13443274 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-0.92 #544 (See [https://builds.apache.org/job/HBase-0.92/544/])
HBASE-6586 Quarantine Corrupted HFiles with hbck (Revision 1378149)
Result = SUCCESS
jmhsieh :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439742#comment-13439742 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542008/hbase-6586-trunk-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 7 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2651//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v7.patch
more spacing fixes and review updates.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434568#comment-13434568 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
Our workaround thus far have been to manually sideline the corrupted hfiles -- check them using 'hbase hfile' and if they aren't read properly, we set them aside. This quarantining behavior is essentially the same but happens automatically.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Description:
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
was:
We've encountered a few upgrades that get stuck from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-94-v3.patch
hbase-6586-92-v3.patch
hbase-6586-trunk-v3.patch
v3, including backports to 92/94.
Minor changes due to HBASE-5714 perms checks.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v8.patch
Removed unused SchemaMetrics code from TestHFile
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439889#comment-13439889 ]
Zhihong Ted Yu commented on HBASE-6586:
---------------------------------------
@Benoit:
What do you think of the proposed exception hierarchy in the patch ?
Thanks
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: hbase-6586-trunk-v6.patch)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439788#comment-13439788 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
interesting -- something on my internal builds didn't run this test to report failure.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436106#comment-13436106 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
Looks like if we go with the hbck approach instead of the autofix approach, we'll should to add yet another state for a region which I'll call something like QUARANTINED. Currently, corrupted HFiles cause a region to go into RS_ZK_REGION_FAILED_OPEN, which eventually gets transitioned to M_ZK_REGION_OFFLINE which then transitions to RS_ZK_REGION_OPENING triggering another attempt to open the region (which fails and ...). Ideally when we have a none-recoverable failure like corrupted hfiles, we'd transition to QUARANTINED instead of FAILED_OPEN and stay there until an admin fixes the problem.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439835#comment-13439835 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542026/hbase-6586-trunk-v4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 7 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2653//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: 0001-hbase-6586-cosmetic-changes-from-review-part-2.patch
fixed all spacing problems and other review concerns.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v6.patch
(previous version had part of another patch i was working on).
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434744#comment-13434744 ]
Andrew Purtell commented on HBASE-6586:
---------------------------------------
I also think it's a bit dangerous to do without admin involvement. As a hbck option this would be great.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434683#comment-13434683 ]
Todd Lipcon commented on HBASE-6586:
------------------------------------
Automatically quarantining the files seems like a dangerous default. Quarantining the region in some way such that all calls to it would fail until the administrator fixes it seems like a better approach. My reasoning is that quarantining an HFile is "silent" data loss (or inconsistency). Data may reappear or revert to an old version. We can't accept that without a user confirming it.
I don't think this is related to HDFS-3731 -- the truncation we saw was not on a block boundary, and that bug would only cause the disappearance of an entire block.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586.patch
Version for trunk. Tested locally, seems to pass. Will post on review board, and after it make it through there I plan to backport to 0.94/0.92.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: hbase-6586.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436113#comment-13436113 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
[~apurtell] I do, I built it yesterday and am polishing up the test currently, and looking for more corner cases. It is a separate phase (it ignores metadata) and is kind of analogous to "checking every block" on a file system. The patch is about 2x the size of the previous "just fix" version.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442894#comment-13442894 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542711/hbase-6586-trunk-v9.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2712//console
This message is automatically generated.
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439797#comment-13439797 ]
Jonathan Hsieh edited comment on HBASE-6586 at 8/23/12 6:49 AM:
----------------------------------------------------------------
v4, changes io exception to illegal argument exception in TestFixedFileTrailer.
was (Author: jmhsieh):
v4, changes io exception to illegal argument exception.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440804#comment-13440804 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542212/hbase-6586-92-v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2676//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: hbase-6586-trunk-v8.patch)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435986#comment-13435986 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
I looped the test and after about 1600 iterations it has not failed. (TestLocaHBaseCluster).
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: 0001-hbase-6586-cosmetic-changes-from-review-part-2.patch)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v8.patch
Doing this again because hadoopqa picked up the wrong patch.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v6.patch
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442620#comment-13442620 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542628/hbase-6586-trunk-v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 106 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2703//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442917#comment-13442917 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-TRUNK #3287 (See [https://builds.apache.org/job/HBase-TRUNK/3287/])
HBASE-6586 Quarantine Corrupted HFiles with hbck (Revision 1377941)
Result = SUCCESS
jmhsieh :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Issue Type: Improvement (was: Bug)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: hbase-6586.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: hbase-6586-trunk-v3.patch)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442941#comment-13442941 ]
Ted Yu commented on HBASE-6586:
-------------------------------
@Jon:
Please go ahead.
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: (was: hbase-6586-trunk-v8.patch)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Patch Available (was: Open)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v5.patch
v5, addresses ted's comments.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Fix Version/s: (was: 0.94.3)
0.94.2
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440090#comment-13440090 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542089/hbase-6586-trunk-v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 5 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2656//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439961#comment-13439961 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542046/hbase-6586-trunk-v5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 6 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2654//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh reassigned HBASE-6586:
-------------------------------------
Assignee: Jonathan Hsieh
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442892#comment-13442892 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
Jimmy found an unnecessary synchronized keyword in the code -- v9 removes it.
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442659#comment-13442659 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542644/hbase-6586-trunk-v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
-1 javadoc. The javadoc tool appears to have generated 106 warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2705//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440901#comment-13440901 ]
Zhihong Ted Yu commented on HBASE-6586:
---------------------------------------
@Jon:
As long as Hadoop QA shows green build (barring flaky tests) and your test doesn't turn up surprises, I am fine with integration.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434617#comment-13434617 ]
Andrew Purtell commented on HBASE-6586:
---------------------------------------
So the quarantine should happen automatically, or is this an additional mode/flag for hbck?
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442575#comment-13442575 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542640/hbase-6586-92-v8b.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2704//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439730#comment-13439730 ]
Hadoop QA commented on HBASE-6586:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542006/hbase-6586-94-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2650//console
This message is automatically generated.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-92-v8.patch
hbase-6586-94-v8.patch
Backported to 92 and 94.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v3.patch
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Attachment: hbase-6586-trunk-v9.patch
hbase-6586-94-v9.patch
hbase-6586-92-v9.patch
Versions I've committed to trunk and 0.94. Also the version I was planning on committing to 0.92 (Ted wants to run the suite before letting the patch in).
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435244#comment-13435244 ]
Todd Lipcon commented on HBASE-6586:
------------------------------------
Can you loop that test until it fails, perhaps? I think getting full logs from a run is necessary to determine if it's an HDFS or HBase bug.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles with hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Summary: Quarantine Corrupted HFiles with hbck (was: Quarantine Corrupted HFiles)
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435986#comment-13435986 ]
Jonathan Hsieh edited comment on HBASE-6586 at 8/17/12 1:12 AM:
----------------------------------------------------------------
I looped the test on 0.92.2 repo and after about 1600 iterations it has not failed. (TestLocaHBaseCluster).
was (Author: jmhsieh):
I looped the test and after about 1600 iterations it has not failed. (TestLocaHBaseCluster).
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435220#comment-13435220 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
The error has shown up here also in a unit test.
http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3CCALte62z-+QGLGD4rr3tXVbHwTWK7aOrvARrsLtbQfbDMDS_59A@mail.gmail.com%3E
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448259#comment-13448259 ]
Hudson commented on HBASE-6586:
-------------------------------
Integrated in HBase-0.94-security #51 (See [https://builds.apache.org/job/HBase-0.94-security/51/])
HBASE-6586 Quarantine Corrputed HFiles with hbck (Revision 1377942)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CorruptHFileException.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles with
hbck
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442937#comment-13442937 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
Ted, you cool with commit?
> Quarantine Corrupted HFiles with hbck
> -------------------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.1
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: 0.92.2, 0.96.0, 0.94.3
>
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8a.patch, hbase-6586-92-v8b.patch, hbase-6586-92-v8.patch, hbase-6586-92-v9.patch, hbase-6586-94-v3.patch, hbase-6586-94-v9.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch, hbase-6586-trunk-v9.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440898#comment-13440898 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
I believe I've addressed the review comments. [~tedyu@apache.org], any more comments? If not can I get a +1 (to got with Jimmy's) or a tentative one unless Benoit chimes in?
[~tsuna], do you have any comments wrt to HBaseIOException? I'm currently testing the backported versions of the patches (trivial changes -- removing Audience, other minor tweaks), and hope to commit over the weekend.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-92-v8.patch, hbase-6586-94-v3.patch, hbase-6586-94-v8.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch, hbase-6586-trunk-v5.patch, hbase-6586-trunk-v6.patch, hbase-6586-trunk-v7.patch, hbase-6586-trunk-v8.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436396#comment-13436396 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
I filed a follow-on issue HBASE-6600 for the QUARANTINED region state.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: hbase-6586.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6586:
----------------------------------
Status: Open (was: Patch Available)
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Improvement
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Attachments: 0001-hbase-6568-hbck-quarantine-v6.patch, hbase-6586-92-v3.patch, hbase-6586-94-v3.patch, hbase-6586.patch, hbase-6586-trunk-v3.patch, hbase-6586-trunk-v4.patch
>
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434643#comment-13434643 ]
Jonathan Hsieh commented on HBASE-6586:
---------------------------------------
I've done ports to 0.92/0.94/trunk where it does it automatically and not part of hbck. Its currently testing on all three branches locally, will post if they come back cleanly.
Maybe we should only do it automatically when a flag is enabled? Alternately, we could adds some metric or scary warning on the ui when this happens, but I'll leave for a follow on jira.
> Quarantine Corrupted HFiles
> ---------------------------
>
> Key: HBASE-6586
> URL: https://issues.apache.org/jira/browse/HBASE-6586
> Project: HBase
> Issue Type: Bug
> Reporter: Jonathan Hsieh
>
> We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92 hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the problem in my dev environment but we suspect this may be related to HDFS-3731. On the HBase side, it seems reasonable to quarantine what are most likely truncated hfiles, so that can could later be recovered.
> Here's an example of the exception we've encountered:
> {code}
> 2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler (OpenRegionHandler.java:openRegion(346)) - Failed open of region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
> 0d.
> java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1026)
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
> at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
> at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
> at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:223)
> at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 842220600 (expected to be between 1 and 2)
> at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
> ... 17 more
> {code}
> Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira