You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2017/01/13 14:30:11 UTC
Re: Region comapction failed
In the second case, the error happened when writing hfile. Can you track down the path of the new file so that further investigation can be done ?
Does the table use any encoding ?
Thanks
> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi,
>
> We met a weird issue in our production environment.
>
> Region compaction is always failing with following errors,
>
> 1.
> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483858654825 | Compaction failed Request = regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, time=1747414906352088 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column actually smaller than the previous column: XXXXXXX
> at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:114)
> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:457)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:551)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:328)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecuto
>
> 2.
> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483686810953 | Compaction failed Request = regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68, time=1748294500157323 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX after XXXXXXXXXXXX
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:911)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:947)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:337)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> HBase version : 1.0.2
>
> We have verified all the HFiles in the store using HFilePrettyPrinter with "k" (checkrow), all report is normal. Full scan is also successful.
> We don't have the access to the actual data and may be customer wont agree to share that .
>
> Have anyone faced this issue, any pointers will be much appreciated.
>
> Thanks & Regards,
> Pankaj
Re: Region comapction failed
Posted by Ted Yu <yu...@gmail.com>.
This is likely a new bug.
Please consider opening a JIRA.
> On Jan 16, 2017, at 2:22 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Yeah, HBASE-13123 already available.
>
> While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't contain complete column_family:qualifier information. Normally in HFIle Keys are formed as,
> K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0
>
> But some records,
> K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0
> Here there is no column family detail, normally it is like column family name and the colon (:) .
>
> It's strange, how key is written in wrong format in HFile.
>
>
> Regards,
> Pankaj
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Saturday, January 14, 2017 9:41 AM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> w.r.t. #2, I did a quick search for bloom related fixes.
>
> I found HBASE-13123 but it was in 1.0.2
>
> Planning to spend more time in the next few days.
>
>> On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
>>
>> Thanks Ted for replying.
>>
>> Actually issue happened in production environment and there are many
>> HFiles in that store (can't get the file). As we don't log the file
>> name which is corrupted, Is there anyway to get the corrupted file name?
>>
>> Block encoding is "NONE", table schema has bloom filter as "ROW",
>> compression type is "Snappy" and durability is SKIP_WAL.
>>
>>
>> Regards,
>> Pankaj
>>
>>
>> -----Original Message-----
>> From: Ted Yu [mailto:yuzhihong@gmail.com]
>> Sent: Friday, January 13, 2017 10:30 PM
>> To: dev@hbase.apache.org
>> Cc: user@hbase.apache.org
>> Subject: Re: Region comapction failed
>>
>> In the second case, the error happened when writing hfile. Can you
>> track down the path of the new file so that further investigation can be done ?
>>
>> Does the table use any encoding ?
>>
>> Thanks
>>
>>> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
>>>
>>> Hi,
>>>
>>> We met a weird issue in our production environment.
>>>
>>> Region compaction is always failing with following errors,
>>>
>>> 1.
>>> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483858654825 | Compaction failed Request =
>> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M,
>> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
>> time=1747414906352088 |
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
>> CompactionRunner.doCompaction(CompactSplitThread.java:562)
>>> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into
>>> a
>> column actually smaller than the previous column: XXXXXXX
>>> at org.apache.hadoop.hbase.regionserver.
>> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
>> java:114)
>>> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>> match(ScanQueryMatcher.java:457)
>>> at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> next(StoreScanner.java:551)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:328)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>> at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> at java.util.concurrent.ThreadPoolExecuto
>>>
>>> 2.
>>> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483686810953 | Compaction failed Request =
>> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9
>> M,
>> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
>> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:562)
>>> java.io.IOException: Non-increasing Bloom keys:
>>> XXXXXXXXXXXXXXXXXXXXXX
>> after XXXXXXXXXXXX
>>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> appendGeneralBloomfilter(StoreFile.java:911)
>>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> append(StoreFile.java:947)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:337)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>> at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> HBase version : 1.0.2
>>>
>>> We have verified all the HFiles in the store using
>>> HFilePrettyPrinter
>> with "k" (checkrow), all report is normal. Full scan is also successful.
>>> We don't have the access to the actual data and may be customer wont
>> agree to share that .
>>>
>>> Have anyone faced this issue, any pointers will be much appreciated.
>>>
>>> Thanks & Regards,
>>> Pankaj
>>
Re: Region comapction failed
Posted by Ted Yu <yu...@gmail.com>.
This is likely a new bug.
Please consider opening a JIRA.
> On Jan 16, 2017, at 2:22 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Yeah, HBASE-13123 already available.
>
> While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't contain complete column_family:qualifier information. Normally in HFIle Keys are formed as,
> K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0
>
> But some records,
> K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0
> Here there is no column family detail, normally it is like column family name and the colon (:) .
>
> It's strange, how key is written in wrong format in HFile.
>
>
> Regards,
> Pankaj
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Saturday, January 14, 2017 9:41 AM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> w.r.t. #2, I did a quick search for bloom related fixes.
>
> I found HBASE-13123 but it was in 1.0.2
>
> Planning to spend more time in the next few days.
>
>> On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
>>
>> Thanks Ted for replying.
>>
>> Actually issue happened in production environment and there are many
>> HFiles in that store (can't get the file). As we don't log the file
>> name which is corrupted, Is there anyway to get the corrupted file name?
>>
>> Block encoding is "NONE", table schema has bloom filter as "ROW",
>> compression type is "Snappy" and durability is SKIP_WAL.
>>
>>
>> Regards,
>> Pankaj
>>
>>
>> -----Original Message-----
>> From: Ted Yu [mailto:yuzhihong@gmail.com]
>> Sent: Friday, January 13, 2017 10:30 PM
>> To: dev@hbase.apache.org
>> Cc: user@hbase.apache.org
>> Subject: Re: Region comapction failed
>>
>> In the second case, the error happened when writing hfile. Can you
>> track down the path of the new file so that further investigation can be done ?
>>
>> Does the table use any encoding ?
>>
>> Thanks
>>
>>> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
>>>
>>> Hi,
>>>
>>> We met a weird issue in our production environment.
>>>
>>> Region compaction is always failing with following errors,
>>>
>>> 1.
>>> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483858654825 | Compaction failed Request =
>> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M,
>> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
>> time=1747414906352088 |
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
>> CompactionRunner.doCompaction(CompactSplitThread.java:562)
>>> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into
>>> a
>> column actually smaller than the previous column: XXXXXXX
>>> at org.apache.hadoop.hbase.regionserver.
>> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
>> java:114)
>>> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
>> match(ScanQueryMatcher.java:457)
>>> at org.apache.hadoop.hbase.regionserver.StoreScanner.
>> next(StoreScanner.java:551)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:328)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>> at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> at java.util.concurrent.ThreadPoolExecuto
>>>
>>> 2.
>>> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
>> PORT-longCompactions-1483686810953 | Compaction failed Request =
>> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9
>> M,
>> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
>> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:562)
>>> java.io.IOException: Non-increasing Bloom keys:
>>> XXXXXXXXXXXXXXXXXXXXXX
>> after XXXXXXXXXXXX
>>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> appendGeneralBloomfilter(StoreFile.java:911)
>>> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
>> append(StoreFile.java:947)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> Compactor.performCompaction(Compactor.java:337)
>>> at org.apache.hadoop.hbase.regionserver.compactions.
>> DefaultCompactor.compact(DefaultCompactor.java:104)
>>> at org.apache.hadoop.hbase.regionserver.
>> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
>> java:133)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HStore.compact(
>> HStore.java:1243)
>>> at
>>> org.apache.hadoop.hbase.regionserver.HRegion.compact(
>> HRegion.java:1895)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.doCompaction(
>> CompactSplitThread.java:546)
>>> at org.apache.hadoop.hbase.regionserver.
>> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> HBase version : 1.0.2
>>>
>>> We have verified all the HFiles in the store using
>>> HFilePrettyPrinter
>> with "k" (checkrow), all report is normal. Full scan is also successful.
>>> We don't have the access to the actual data and may be customer wont
>> agree to share that .
>>>
>>> Have anyone faced this issue, any pointers will be much appreciated.
>>>
>>> Thanks & Regards,
>>> Pankaj
>>
RE: Region comapction failed
Posted by Pankaj kr <pa...@huawei.com>.
Yeah, HBASE-13123 already available.
While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't contain complete column_family:qualifier information. Normally in HFIle Keys are formed as,
K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0
But some records,
K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0
Here there is no column family detail, normally it is like column family name and the colon (:) .
It's strange, how key is written in wrong format in HFile.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Saturday, January 14, 2017 9:41 AM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: Region comapction failed
w.r.t. #2, I did a quick search for bloom related fixes.
I found HBASE-13123 but it was in 1.0.2
Planning to spend more time in the next few days.
On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
> Thanks Ted for replying.
>
> Actually issue happened in production environment and there are many
> HFiles in that store (can't get the file). As we don't log the file
> name which is corrupted, Is there anyway to get the corrupted file name?
>
> Block encoding is "NONE", table schema has bloom filter as "ROW",
> compression type is "Snappy" and durability is SKIP_WAL.
>
>
> Regards,
> Pankaj
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, January 13, 2017 10:30 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> In the second case, the error happened when writing hfile. Can you
> track down the path of the new file so that further investigation can be done ?
>
> Does the table use any encoding ?
>
> Thanks
>
> > On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
> >
> > Hi,
> >
> > We met a weird issue in our production environment.
> >
> > Region compaction is always failing with following errors,
> >
> > 1.
> > 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483858654825 | Compaction failed Request =
> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M,
> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
> time=1747414906352088 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
> CompactionRunner.doCompaction(CompactSplitThread.java:562)
> > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into
> > a
> column actually smaller than the previous column: XXXXXXX
> > at org.apache.hadoop.hbase.regionserver.
> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
> java:114)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:457)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:551)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:328)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at
> > org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecuto
> >
> > 2.
> > 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483686810953 | Compaction failed Request =
> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9
> M,
> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:562)
> > java.io.IOException: Non-increasing Bloom keys:
> > XXXXXXXXXXXXXXXXXXXXXX
> after XXXXXXXXXXXX
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> appendGeneralBloomfilter(StoreFile.java:911)
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> append(StoreFile.java:947)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:337)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at
> > org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > HBase version : 1.0.2
> >
> > We have verified all the HFiles in the store using
> > HFilePrettyPrinter
> with "k" (checkrow), all report is normal. Full scan is also successful.
> > We don't have the access to the actual data and may be customer wont
> agree to share that .
> >
> > Have anyone faced this issue, any pointers will be much appreciated.
> >
> > Thanks & Regards,
> > Pankaj
>
RE: Region comapction failed
Posted by Pankaj kr <pa...@huawei.com>.
Yeah, HBASE-13123 already available.
While going through HFilePrettyPrinter result with -e option, we observed some keys doesn't contain complete column_family:qualifier information. Normally in HFIle Keys are formed as,
K: r1/CF:Qualifier/1484302946516/Put/vlen=1/seqid=0
But some records,
K: r1/Unknown_Qualifier/1484302946516/Put/vlen=1/seqid=0
Here there is no column family detail, normally it is like column family name and the colon (:) .
It's strange, how key is written in wrong format in HFile.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Saturday, January 14, 2017 9:41 AM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: Region comapction failed
w.r.t. #2, I did a quick search for bloom related fixes.
I found HBASE-13123 but it was in 1.0.2
Planning to spend more time in the next few days.
On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
> Thanks Ted for replying.
>
> Actually issue happened in production environment and there are many
> HFiles in that store (can't get the file). As we don't log the file
> name which is corrupted, Is there anyway to get the corrupted file name?
>
> Block encoding is "NONE", table schema has bloom filter as "ROW",
> compression type is "Snappy" and durability is SKIP_WAL.
>
>
> Regards,
> Pankaj
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, January 13, 2017 10:30 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> In the second case, the error happened when writing hfile. Can you
> track down the path of the new file so that further investigation can be done ?
>
> Does the table use any encoding ?
>
> Thanks
>
> > On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
> >
> > Hi,
> >
> > We met a weird issue in our production environment.
> >
> > Region compaction is always failing with following errors,
> >
> > 1.
> > 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483858654825 | Compaction failed Request =
> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M,
> 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5,
> time=1747414906352088 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
> CompactionRunner.doCompaction(CompactSplitThread.java:562)
> > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into
> > a
> column actually smaller than the previous column: XXXXXXX
> > at org.apache.hadoop.hbase.regionserver.
> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
> java:114)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:457)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:551)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:328)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at
> > org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecuto
> >
> > 2.
> > 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483686810953 | Compaction failed Request =
> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9
> M,
> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:562)
> > java.io.IOException: Non-increasing Bloom keys:
> > XXXXXXXXXXXXXXXXXXXXXX
> after XXXXXXXXXXXX
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> appendGeneralBloomfilter(StoreFile.java:911)
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> append(StoreFile.java:947)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:337)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at
> > org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > HBase version : 1.0.2
> >
> > We have verified all the HFiles in the store using
> > HFilePrettyPrinter
> with "k" (checkrow), all report is normal. Full scan is also successful.
> > We don't have the access to the actual data and may be customer wont
> agree to share that .
> >
> > Have anyone faced this issue, any pointers will be much appreciated.
> >
> > Thanks & Regards,
> > Pankaj
>
Re: Region comapction failed
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. #2, I did a quick search for bloom related fixes.
I found HBASE-13123 but it was in 1.0.2
Planning to spend more time in the next few days.
On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
> Thanks Ted for replying.
>
> Actually issue happened in production environment and there are many
> HFiles in that store (can't get the file). As we don't log the file name
> which is corrupted, Is there anyway to get the corrupted file name?
>
> Block encoding is "NONE", table schema has bloom filter as "ROW",
> compression type is "Snappy" and durability is SKIP_WAL.
>
>
> Regards,
> Pankaj
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, January 13, 2017 10:30 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> In the second case, the error happened when writing hfile. Can you track
> down the path of the new file so that further investigation can be done ?
>
> Does the table use any encoding ?
>
> Thanks
>
> > On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
> >
> > Hi,
> >
> > We met a weird issue in our production environment.
> >
> > Region compaction is always failing with following errors,
> >
> > 1.
> > 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483858654825 | Compaction failed Request =
> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 20.8
> M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, time=1747414906352088 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
> CompactionRunner.doCompaction(CompactSplitThread.java:562)
> > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a
> column actually smaller than the previous column: XXXXXXX
> > at org.apache.hadoop.hbase.regionserver.
> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
> java:114)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:457)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:551)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:328)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecuto
> >
> > 2.
> > 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483686810953 | Compaction failed Request =
> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 M,
> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:562)
> > java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX
> after XXXXXXXXXXXX
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> appendGeneralBloomfilter(StoreFile.java:911)
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> append(StoreFile.java:947)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:337)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > HBase version : 1.0.2
> >
> > We have verified all the HFiles in the store using HFilePrettyPrinter
> with "k" (checkrow), all report is normal. Full scan is also successful.
> > We don't have the access to the actual data and may be customer wont
> agree to share that .
> >
> > Have anyone faced this issue, any pointers will be much appreciated.
> >
> > Thanks & Regards,
> > Pankaj
>
Re: Region comapction failed
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. #2, I did a quick search for bloom related fixes.
I found HBASE-13123 but it was in 1.0.2
Planning to spend more time in the next few days.
On Fri, Jan 13, 2017 at 5:29 PM, Pankaj kr <pa...@huawei.com> wrote:
> Thanks Ted for replying.
>
> Actually issue happened in production environment and there are many
> HFiles in that store (can't get the file). As we don't log the file name
> which is corrupted, Is there anyway to get the corrupted file name?
>
> Block encoding is "NONE", table schema has bloom filter as "ROW",
> compression type is "Snappy" and durability is SKIP_WAL.
>
>
> Regards,
> Pankaj
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, January 13, 2017 10:30 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: Region comapction failed
>
> In the second case, the error happened when writing hfile. Can you track
> down the path of the new file so that further investigation can be done ?
>
> Does the table use any encoding ?
>
> Thanks
>
> > On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
> >
> > Hi,
> >
> > We met a weird issue in our production environment.
> >
> > Region compaction is always failing with following errors,
> >
> > 1.
> > 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483858654825 | Compaction failed Request =
> regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 20.8
> M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, time=1747414906352088 |
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$
> CompactionRunner.doCompaction(CompactSplitThread.java:562)
> > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a
> column actually smaller than the previous column: XXXXXXX
> > at org.apache.hadoop.hbase.regionserver.
> ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.
> java:114)
> > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.
> match(ScanQueryMatcher.java:457)
> > at org.apache.hadoop.hbase.regionserver.StoreScanner.
> next(StoreScanner.java:551)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:328)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecuto
> >
> > 2.
> > 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:
> PORT-longCompactions-1483686810953 | Compaction failed Request =
> regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 M,
> 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68,
> time=1748294500157323 | org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:562)
> > java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX
> after XXXXXXXXXXXX
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> appendGeneralBloomfilter(StoreFile.java:911)
> > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.
> append(StoreFile.java:947)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> Compactor.performCompaction(Compactor.java:337)
> > at org.apache.hadoop.hbase.regionserver.compactions.
> DefaultCompactor.compact(DefaultCompactor.java:104)
> > at org.apache.hadoop.hbase.regionserver.
> DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.
> java:133)
> > at org.apache.hadoop.hbase.regionserver.HStore.compact(
> HStore.java:1243)
> > at org.apache.hadoop.hbase.regionserver.HRegion.compact(
> HRegion.java:1895)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.doCompaction(
> CompactSplitThread.java:546)
> > at org.apache.hadoop.hbase.regionserver.
> CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > HBase version : 1.0.2
> >
> > We have verified all the HFiles in the store using HFilePrettyPrinter
> with "k" (checkrow), all report is normal. Full scan is also successful.
> > We don't have the access to the actual data and may be customer wont
> agree to share that .
> >
> > Have anyone faced this issue, any pointers will be much appreciated.
> >
> > Thanks & Regards,
> > Pankaj
>
RE: Region comapction failed
Posted by Pankaj kr <pa...@huawei.com>.
Thanks Ted for replying.
Actually issue happened in production environment and there are many HFiles in that store (can't get the file). As we don't log the file name which is corrupted, Is there anyway to get the corrupted file name?
Block encoding is "NONE", table schema has bloom filter as "ROW", compression type is "Snappy" and durability is SKIP_WAL.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 13, 2017 10:30 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: Region comapction failed
In the second case, the error happened when writing hfile. Can you track down the path of the new file so that further investigation can be done ?
Does the table use any encoding ?
Thanks
> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi,
>
> We met a weird issue in our production environment.
>
> Region compaction is always failing with following errors,
>
> 1.
> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483858654825 | Compaction failed Request = regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, time=1747414906352088 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column actually smaller than the previous column: XXXXXXX
> at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:114)
> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:457)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:551)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:328)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecuto
>
> 2.
> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483686810953 | Compaction failed Request = regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68, time=1748294500157323 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX after XXXXXXXXXXXX
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:911)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:947)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:337)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> HBase version : 1.0.2
>
> We have verified all the HFiles in the store using HFilePrettyPrinter with "k" (checkrow), all report is normal. Full scan is also successful.
> We don't have the access to the actual data and may be customer wont agree to share that .
>
> Have anyone faced this issue, any pointers will be much appreciated.
>
> Thanks & Regards,
> Pankaj
RE: Region comapction failed
Posted by Pankaj kr <pa...@huawei.com>.
Thanks Ted for replying.
Actually issue happened in production environment and there are many HFiles in that store (can't get the file). As we don't log the file name which is corrupted, Is there anyway to get the corrupted file name?
Block encoding is "NONE", table schema has bloom filter as "ROW", compression type is "Snappy" and durability is SKIP_WAL.
Regards,
Pankaj
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, January 13, 2017 10:30 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: Region comapction failed
In the second case, the error happened when writing hfile. Can you track down the path of the new file so that further investigation can be done ?
Does the table use any encoding ?
Thanks
> On Jan 13, 2017, at 2:47 AM, Pankaj kr <pa...@huawei.com> wrote:
>
> Hi,
>
> We met a weird issue in our production environment.
>
> Region compaction is always failing with following errors,
>
> 1.
> 2017-01-10 02:19:10,427 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483858654825 | Compaction failed Request = regionName=XXXX., storeName=XYZ, fileCount=6, fileSize=100.7 M (3.2 M, 20.8 M, 15.1 M, 20.9 M, 21.0 M, 19.7 M), priority=-5, time=1747414906352088 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column actually smaller than the previous column: XXXXXXX
> at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:114)
> at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:457)
> at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:551)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:328)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecuto
>
> 2.
> 2017-01-10 02:33:53,009 | ERROR | regionserver/RS-HOST/RS-IP:PORT-longCompactions-1483686810953 | Compaction failed Request = regionName=YYYYYY, storeName=ABC, fileCount=6, fileSize=125.3 M (20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M, 20.9 M), priority=-68, time=1748294500157323 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:562)
> java.io.IOException: Non-increasing Bloom keys: XXXXXXXXXXXXXXXXXXXXXX after XXXXXXXXXXXX
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:911)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:947)
> at org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:337)
> at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:104)
> at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:133)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1243)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1895)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:546)
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:583)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> HBase version : 1.0.2
>
> We have verified all the HFiles in the store using HFilePrettyPrinter with "k" (checkrow), all report is normal. Full scan is also successful.
> We don't have the access to the actual data and may be customer wont agree to share that .
>
> Have anyone faced this issue, any pointers will be much appreciated.
>
> Thanks & Regards,
> Pankaj