You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Toshihiro Suzuki (JIRA)" <ji...@apache.org> on 2018/10/05 15:55:00 UTC
[jira] [Comment Edited] (HBASE-21200) Memstore flush doesn't finish
because of seekToPreviousRow() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639976#comment-16639976 ]
Toshihiro Suzuki edited comment on HBASE-21200 at 10/5/18 3:54 PM:
-------------------------------------------------------------------
Thank you for reviewing [~ram_krish].
{quote}
So before this patch - the same test runs successfully is it? So the cells greater than the readPt getting skipped inside the seek() call I believe.
{quote}
Yes, the same test runs successfully before applying this patch. it's just a optimization for speed.
{quote}
Even there we were doing getIterator() and iterating through the elements and comparing the seqId with the readPt.
So exactly how does the speed up happen here?
{quote}
Yes, we were doing getIterator() and iterating through the elements and comparing the seqId with the readPt. In original logic in seekToPreviousRow(), we seek row while going backward the row until finding right previous cell whose seqId is less than or equal to readPt. When after opening scanner we put a lot of cells whose seqId is greater than the readPt of the scanner to the memstore, a lot of seeking row will happen as described in the following image in HBASE-15871:
https://issues.apache.org/jira/secure/attachment/12805207/memstore_backwardSeek%28%29.PNG
This causes that seek() repeatedly search cells that are already searched, and it's very slow.
In the optimization I added in the patch, we seek row only when the seqId of any of the cells of the row is less than or equal to the readPt. We can do this because if seqIds of all the cells of row are greater than the readPt, I think we won't find right previous cell when seeking the row. So we can skip seeking row in this case.
Thanks.
was (Author: brfrn169):
Thank you for reviewing [~ram_krish].
{quote}
So before this patch - the same test runs successfully is it? So the cells greater than the readPt getting skipped inside the seek() call I believe.
{quote}
Yes, the same test runs successfully before applying this patch. it's just a optimization for speed.
{quote}
Even there we were doing getIterator() and iterating through the elements and comparing the seqId with the readPt.
So exactly how does the speed up happen here?
{quote}
Yes, we were doing getIterator() and iterating through the elements and comparing the seqId with the readPt. In original logic in seekToPreviousRow(), we seek row while going backward the row until finding right previous cell whose seqId is less than or equal to readPt. When after opening scanner we put a lot of cells whose seqId is greater than the readPt of the scanner to the memstore, a lot of seeking row will happen as described in the following image in HBASE-15871:
https://issues.apache.org/jira/secure/attachment/12805207/memstore_backwardSeek%28%29.PNG
This causes that seek() repeatedly search cells that are already searched, and it's very slow.
In the optimization I added in the patch, we seek row only when the seqId of any of the cells of the row is less than or equal to the readPt. We can do this because if seqIds of all the cells are greater than the readPt, we won't find right previous cell. So we can skip seeking row in this case.
Thanks.
> Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.
> ---------------------------------------------------------------------------------
>
> Key: HBASE-21200
> URL: https://issues.apache.org/jira/browse/HBASE-21200
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Reporter: dongjin2193.jeon
> Assignee: Toshihiro Suzuki
> Priority: Major
> Attachments: HBASE-21200-UT.patch, HBASE-21200.master.001.patch, RegionServerJstack.log
>
>
> The issue of delaying memstore flush still occurs after backport hbase-15871.
> Reverse scan takes a long time to seek previous row in the memstore full of deleted cells.
>
> jstack :
> "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x00007fa3d0729000 nid=0x486a waiting on condition [0x00007fa3b9b6b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000a465fe60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)*
> at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127)
> at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106)
> at org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130)
> at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256)
> at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110)
> at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:748)
>
> "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5 os_prio=0 tid=0x00007fa3e6280000 nid=0x4801 runnable [0x00007fa3bd29a000]
> java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780)
> at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826)
> - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818)
> - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000)
> - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
> at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136)
> at org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)*
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> - locked <0x000000009ee8ca10> (a org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)