You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Subash Kunjupillai <su...@ericsson.com> on 2017/09/23 15:25:54 UTC

Regions not coming online because HFile invalid version error

We have been loading data in our HBase and we observed that few regions were
in FAILED_OPEN state. On going through few suggestions, I tried to run hbck
with no success of fixing the region. Then tried by deleting the WAL
directory from HDFS. Now I'm able to see the below error in RegionServer
log.

2017-09-23 16:31:40,660 INFO  [RS_OPEN_REGION-dl360x2923:16020-2]
coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
54b0bce397eefb930bce5767513220d7, NAME =>
'event_2017-01-17,eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148266980.54b0bce397eefb930bce5767513220d7.',
STARTKEY =>
'eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
ENDKEY =>
'f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'}
failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
7978
2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
regionserver.HRegion: Could not initialize all stores for the
region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.
2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
handler.OpenRegionHandler: Failed open of
region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
    at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:935)
    at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:823)
    at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:798)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6299)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6260)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6231)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6187)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6138)
    at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
    at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
    at
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:556)
    at
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:511)
    at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:273)
    at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:4980)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:909)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:906)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
    at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477)
    at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505)
    at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1066)
    at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:251)
    at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:394)
    at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:495)
    at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:661)
    at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:129)
    at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531)
    at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:528)
    ... 6 more
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 0
(expected to be between 2 and 3)
    at
org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:852)
    at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:402)
    at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462)
    ... 15 more
2017-09-23 16:31:41,134 INFO  [RS_OPEN_REGION-dl360x2923:16020-0]
coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
16ccc4ba3f32254e418e8a5fdc69178d, NAME =>
'event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.',
STARTKEY =>
'\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
ENDKEY =>
'\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'}
failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
7978

We are using HBase 1.1.8. Can someone help me on how to overcome this issue.




--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html

Re: Regions not coming online because HFile invalid version error

Posted by Ted Yu <yu...@gmail.com>.
Sidelining means moving the file(s) to a location outside the column family
directory so that the region can be opened.

There have been many releases after 1.1.8
Consider upgrading.

If you can describe more about your application, that would help.

On Wed, Oct 4, 2017 at 9:40 AM, Subash Kunjupillai <su...@ericsson.com>
wrote:

> Sorry for the delayed reply, got stuck in another issue!
>
> There are so many regions in this state. Sorry, I couldn't understand what
> you mean by sidelining. Can you please help me to understand and ways to do
> it.
>
> Data loading was done in normal way through our application. Yes, this
> happened recently in HBase 1.1.8 version.
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>

Re: Regions not coming online because HFile invalid version error

Posted by Subash Kunjupillai <su...@ericsson.com>.
Sorry for the delayed reply, got stuck in another issue!

There are so many regions in this state. Sorry, I couldn't understand what
you mean by sidelining. Can you please help me to understand and ways to do
it.

Data loading was done in normal way through our application. Yes, this
happened recently in HBase 1.1.8 version.



--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html

Re: Regions not coming online because HFile invalid version error

Posted by Ted Yu <yu...@gmail.com>.
bq. Invalid HFile version: 0

The hfile, 77a0546b73d04dffa02597255aff9a08, is corrupt.
Consider sidelining it and continue.

How was the loading done ?
Did this problem happen only recently ?

Cheers

On Sat, Sep 23, 2017 at 8:25 AM, Subash Kunjupillai <su...@ericsson.com>
wrote:

> We have been loading data in our HBase and we observed that few regions
> were
> in FAILED_OPEN state. On going through few suggestions, I tried to run hbck
> with no success of fixing the region. Then tried by deleting the WAL
> directory from HDFS. Now I'm able to see the below error in RegionServer
> log.
>
> 2017-09-23 16:31:40,660 INFO  [RS_OPEN_REGION-dl360x2923:16020-2]
> coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
> 54b0bce397eefb930bce5767513220d7, NAME =>
> 'event_2017-01-17,eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00,1505148266980.
> 54b0bce397eefb930bce5767513220d7.',
> STARTKEY =>
> 'eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
> ENDKEY =>
> 'f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00'}
> failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
> 7978
> 2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
> regionserver.HRegion: Could not initialize all stores for the
> region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.
> 16ccc4ba3f32254e418e8a5fdc69178d.
> 2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
> handler.OpenRegionHandler: Failed open of
> region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.
> 16ccc4ba3f32254e418e8a5fdc69178d.,
> starting to roll back the global memstore size.
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
> maprfs:///hbase/data/default/event_2017-01-02/
> 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(
> HRegion.java:935)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(
> HRegion.java:823)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:798)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.
> openHRegion(HRegion.java:6299)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.
> openHRegion(HRegion.java:6260)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.
> openHRegion(HRegion.java:6231)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.
> openHRegion(HRegion.java:6187)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.
> openHRegion(HRegion.java:6138)
>     at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(
> OpenRegionHandler.java:362)
>     at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(
> OpenRegionHandler.java:129)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
> maprfs:///hbase/data/default/event_2017-01-02/
> 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
>     at
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:
> 556)
>     at
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:
> 511)
>     at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:273)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.
> java:4980)
>     at org.apache.hadoop.hbase.regionserver.HRegion$1.call(
> HRegion.java:909)
>     at org.apache.hadoop.hbase.regionserver.HRegion$1.call(
> HRegion.java:906)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     ... 3 more
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file
> maprfs:///hbase/data/default/event_2017-01-02/
> 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477)
>     at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505)
>     at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.
> <init>(StoreFile.java:1066)
>     at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.
> open(StoreFileInfo.java:251)
>     at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:394)
>     at
> org.apache.hadoop.hbase.regionserver.StoreFile.
> createReader(StoreFile.java:495)
>     at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(
> HStore.java:661)
>     at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:129)
>     at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531)
>     at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:528)
>     ... 6 more
> Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 0
> (expected to be between 2 and 3)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:852)
>     at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(
> FixedFileTrailer.java:402)
>     at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462)
>     ... 15 more
> 2017-09-23 16:31:41,134 INFO  [RS_OPEN_REGION-dl360x2923:16020-0]
> coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
> 16ccc4ba3f32254e418e8a5fdc69178d, NAME =>
> 'event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.
> 16ccc4ba3f32254e418e8a5fdc69178d.',
> STARTKEY =>
> '\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00',
> ENDKEY =>
> '\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00'}
> failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
> 7978
>
> We are using HBase 1.1.8. Can someone help me on how to overcome this
> issue.
>
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>