You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Toshihiro Suzuki (JIRA)" <ji...@apache.org> on 2019/05/27 22:10:00 UTC

[jira] [Commented] (HBASE-21665) OfflineMetaRepair tool fails with NPE

    [ https://issues.apache.org/jira/browse/HBASE-21665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849239#comment-16849239 ] 

Toshihiro Suzuki commented on HBASE-21665:
------------------------------------------

IMO, we need OfflineMetaRepair for HBase-2.x because we don't have a straightforward way to rebuild meta for now.

When I faced table holes in a customer's cluster (HDP-3.x), I needed to do the following steps as a workaround, which was very time-consuming:
1. Backup the table data in HDFS level
2. Drop the problematic table
3. Re-create the table with the previous splits
4. Bulkload the table data from the backup (which is taken by step 1)

Ideally, HBCK2 should have the feature to rebuild meta, but it looks like that's taking more time to do this, so I think we should apply the fix to branch-2 and branch-2.x (and master) for now. And then we will be able to use OfflineMetaRepair as a workaround in case we have holes and overlaps until HBCK2 have the feature.

[~stack] [~busbey] [~elserj]



> OfflineMetaRepair tool fails with NPE
> -------------------------------------
>
>                 Key: HBASE-21665
>                 URL: https://issues.apache.org/jira/browse/HBASE-21665
>             Project: HBase
>          Issue Type: Bug
>          Components: hbck
>    Affects Versions: 2.1.0, 2.0.0
>            Reporter: Pankaj Kumar
>            Assignee: Pankaj Kumar
>            Priority: Critical
>             Fix For: 3.0.0
>
>         Attachments: HBASE-21665.patch
>
>
> OfflineMetaRepair fails with NPE, execute below command
> hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
>  
> {noformat}
> 2019-01-02 16:22:56,387 INFO [main] regionserver.HRegion: Opened 1588230740; next sequenceid=28
> 2019-01-02 16:22:56,459 ERROR [main] hbck.OfflineMetaRepair: Bailed out due to: 
> java.lang.NullPointerException
>  at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.getOrMakeChunk(MemStoreLABImpl.java:335)
>  at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.copyCellInto(MemStoreLABImpl.java:193)
>  at org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.copyCellInto(MemStoreLABImpl.java:115)
>  at org.apache.hadoop.hbase.regionserver.Segment.maybeCloneWithAllocator(Segment.java:183)
>  at org.apache.hadoop.hbase.regionserver.AbstractMemStore.maybeCloneWithAllocator(AbstractMemStore.java:334)
>  at org.apache.hadoop.hbase.regionserver.AbstractMemStore.doAdd(AbstractMemStore.java:157)
>  at org.apache.hadoop.hbase.regionserver.AbstractMemStore.doAddOrUpsert(AbstractMemStore.java:147)
>  at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:117)
>  at org.apache.hadoop.hbase.regionserver.AbstractMemStore.add(AbstractMemStore.java:111)
>  at org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:750)
>  at org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore(HRegion.java:4435)
>  at org.apache.hadoop.hbase.regionserver.HRegion.access$500(HRegion.java:228)
>  at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore(HRegion.java:3495)
>  at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0(HRegion.java:3186)
>  at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3119)
>  at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3178)
>  at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3660)
>  at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4073)
>  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4006)
>  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3937)
>  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3928)
>  at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:1665)
>  at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:121)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)