You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kang Minwoo <mi...@outlook.com> on 2020/02/03 05:21:59 UTC
Re: Extremely long flush times
I didn't upgrade the HBase version or applying the patch.
Because the patch is too big and I have a plan to make a new HBase Cluster.
At that time I consider upgrade the HBase version. (Current HBase version is 1.2.9)
Now I made a Coprocessor that is deleted TTL expired data instead of sending delete requests.
Best regards,
Minwoo Kang
________________________________________
보낸 사람: ramkrishna vasudevan <ra...@gmail.com>
보낸 날짜: 2020년 1월 30일 목요일 14:07
받는 사람: Kang Minwoo
참조: Hbase-User; Stack
제목: Re: Extremely long flush times
Hi Minwoo Kang
Any updates here? Where you able to over come the issue with the upgrade? Or by applying the patch?
Regards
Ram
On Fri, Jan 10, 2020 at 11:44 AM Kang Minwoo <mi...@outlook.com>> wrote:
Thanks for the reply.
It is a lot of help to me.
Best regards,
Minwoo Kang
________________________________________
보낸 사람: ramkrishna vasudevan <ra...@gmail.com>>
보낸 날짜: 2020년 1월 10일 금요일 14:35
받는 사람: Hbase-User
참조: Stack
제목: Re: Extremely long flush times
Hi
In your case you have large compactions going on and at the same time heavy
reads happening. Since there are lot of deletes the scan is spending
sufficient time in file reads.
Since compactions/flushes happens every now and then the readers are
getting reset and that is causing the lock to be acquired and since there
are multiple threads competing your scans suffer more because they are not
able to reset themselves.
Yes - if the above said case is true for your scenario - then HBASE-13082
will help you out. Since it avoid scanners being reset on compactions and
only one flushes and that too it is not a hard call to reset. If the
scanner finds the boolean to be set then it resets if not the scan just
goes on .
Regards
Ram
On Fri, Jan 10, 2020 at 10:01 AM Kang Minwoo <mi...@outlook.com>>
wrote:
> Thank you for reply.
>
> All Regions or just the one?
> => just one
>
> Do thread dumps lock thread reading against hdfs every time you take one?
> => yes
>
> Is it always inside in updateReaders? Is there a bad file or lots of files
> to add to the list?
> => always inside in updateReaders.
>
> ----
>
> Sorry for the delay in reply.
>
> I had to handle this issue.
> Temporarily, I fixed my code that does not occur in that situation that is
> read worthlessness cell.
> After that, The issue hasn't occurred.
>
> Background:
> My application deletes out of date data every day.
> And Region is extremely big. Major compaction spent a lot of time.
> tombstone cell remains a long time.
> If the client read full data. there is a lot of worthlessness cells.
> I think it is a reason for lock thread reading hdfs files.
>
> I'm looking at the HBASE-13082[1].
> (I am not sure HBASE-13082 is related.)
>
> [1]: https://issues.apache.org/jira/browse/HBASE-13082
>
> Best regards,
> Minwoo Kang
>
> ________________________________________
> 보낸 사람: Stack <st...@duboce.net>>
> 보낸 날짜: 2020년 1월 4일 토요일 03:40
> 받는 사람: Hbase-User
> 제목: Re: Extremely long flush times
>
> All Regions or just the one?
>
> Do thread dumps lock thread reading against hdfs every time you take one?
>
> Is it always inside in updateReaders? Is there a bad file or lots of files
> to add to the list?
>
> Yours,
> S
>
>
>
> On Thu, Jan 2, 2020 at 8:34 PM Kang Minwoo <mi...@outlook.com>>
> wrote:
>
> > Hello Users,
> >
> > I met an issue that is flush times is too long.
> >
> > MemStoreFlusher is waiting for a lock.
> > ```
> > "MemStoreFlusher.0"
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x00007f0412bddcb8> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > at
> >
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> > at
> > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:692)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1100)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1079)
> > at
> > org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:118)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2321)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2430)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2153)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2115)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2005)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1930)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:475)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
> > at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:263)
> > at java.lang.Thread.run(Thread.java:748)
> > Locked ownable synchronizers:
> > - None
> > ```
> >
> >
> > RPC Handler had the lock.
> > ```
> > "B.defaultRpcServer.handler"
> > java.lang.Thread.State: RUNNABLE
> > at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
> > at org.apache.log4j.Category.isEnabledFor(Category.java:751)
> > at
> >
> org.apache.commons.logging.impl.Log4JLogger.isTraceEnabled(Log4JLogger.java:303)
> > at
> > org.apache.hadoop.hdfs.BlockReaderLocal.skip(BlockReaderLocal.java:622)
> > - locked <0x00007f0271bcc1c0> (a
> > org.apache.hadoop.hdfs.BlockReaderLocal)
> > at
> > org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1534)
> > - locked <0x00007f038265f980> (a
> > org.apache.hadoop.hdfs.DFSInputStream)
> > at
> > org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileUtil.seekOnMultipleSources(HFileUtil.java:38)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1431)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1654)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1538)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:452)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475)
> > at
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761)
> > at
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622)
> > - locked <0x00007f035a4df0f8> (a
> > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770)
> > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > at java.lang.Thread.run(Thread.java:748)
> > Locked ownable synchronizers:
> > - <0x00007f02bfaf7a90> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > - <0x00007f0412bddcb8> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > ```
> >
> > ```
> > B.defaultRpcServer.handler
> > java.lang.Thread.State: RUNNABLE
> > at java.io.FilterInputStream.read(FilterInputStream.java:83)
> > at
> >
> org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:145)
> > at
> >
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:74)
> > at
> >
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> > at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> > - locked <0x00007f024acf0bf8> (a java.io.BufferedInputStream)
> > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .compress.Compression.decompress(Compression.java:453)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileBlock.unpack(HFileBlock.java:549)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2.readBlock(HFileReaderV2.java:455)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:729)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:854)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> >
> .hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:849)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:866)
> > at org.apache.hadoop.hbase.io<http://org.apache.hadoop.hbase.io>
> > .hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:886)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.skipKVsNewerThanReadpoint(StoreFileScanner.java:247)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:194)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:350)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.resetScannerStack(StoreScanner.java:750)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkReseek(StoreScanner.java:725)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:475)
> > at
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5824)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5994)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5761)
> > at
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2622)
> > - locked <0x00007f035a4df0f8> (a
> > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33770)
> > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2216)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > at java.lang.Thread.run(Thread.java:748)
> > Locked ownable synchronizers:
> > - <0x00007f0412bddcb8> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > ```
> >
> > Versions:
> > - HBase: 1.2.9
> > - HDFS: 2.7.7
> >
> > Best regards,
> > Minwoo Kang
> >
>