[jira] [Comment Edited] (HBASE-6561) Gets/Puts with many column send the RegionServer into an "endless" loop

["Lars Hofhansl (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432841#comment-13432841 ] 

Lars Hofhansl edited comment on HBASE-6561 at 8/13/12 8:37 AM:
---------------------------------------------------------------

When I instrument the code slightly further I see that MemStoreScanner.getNext(...) has to skip over a large number of KVs with a higher memstoreTS (many millions), which causes the complete unresponsiveness of the RS (I tried to let the RS run for a while after I stopped the Puts/Gets from the client, but after 30mins I just killed it).

As a test I changed ScanWildcardColumnTracker.checkVersion to return MatchCode.SKIP instead of MatchCode.SEEK_NEXT_COL (when the max number of versions is reached); and I do not see this behavior.

It seems the problem here stems from the excessive reseeking in this case. The StoreScanner is smart and instructs its KeyValueHeap to seek to the next column (which can potentially skip many columns). Only in this case, this is in fact not so smart, because reseek needs to reset the current iterators, and hence all KV with higher memstoreTS have to be searched again (see also comment in MemStoreScanner.reseek).

There's is no way (that I found) to isolate this issue before it happens.
I have a patch where in reseek the MemStoreScanner opportunistically iterates "a bit" (1000 iterations in my patch), and only then actually reseeks forward.
This eliminates the problem for me.
I have no good numbers of when it is more efficient to seek ahead (and skip many versions, columns, or rows) (i.e. how many iterations the MemStoreScanner should attempt before seeking).

Edit: Spelling.
                
      was (Author: lhofhansl):
    When I instrument the code slight further I see that MemStoreScanner.getNext(...) has to skip over a large number of KVs with a higher memstoreTS (many millions), which causes the complete unresponsiveness of the RS (I tried that let the RS running for a while after I stopped the Puts/Gets from the client, but after 30mins I just killed it).

As a test I changed ScanWildcardColumnTracker.checkVersion to return MatchCode.SKIP instead of MatchCode.SEEK_NEXT_COL (when the max number of versions is reached); and I do not see this behavior.

It seems the problem here stems from the excessive reseeking in this case. The StoreScanner is smart is smart and instructs its KeyValueHeap to seek to the next column (which can potentially skip many columns). Only in this case, this is in fact not so smart, because reseek need to reset the current iterators, and hence all KV with higher memstoreTS have to be searched again (see also comment in MemStoreScanner.reseek).

There's is no way (that I found) to isolate this issue before it happens.
I have a patch where in reseek the MemStoreScanner opportunistically iterates "a bit" (1000 iterations in my patch), and only then actually reseeks forward.
This eliminates the problem for me.
I have no good numbers of when it is more efficient to seek ahead (and skip many versions, columns, or rows) (i.e. how many iterations the MemStoreScanner should attempt before seeking).

                  
> Gets/Puts with many column send the RegionServer into an "endless" loop
> -----------------------------------------------------------------------
>
>                 Key: HBASE-6561
>                 URL: https://issues.apache.org/jira/browse/HBASE-6561
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>             Fix For: 0.96.0, 0.94.2
>
>
> This came from the mailing this:
> We were able to replicate this behavior in a pseudo-distributed hbase
> (hbase-0.94.1) environment. We wrote a test program that creates a test
> table "MyTestTable" and populates it with random rows, then it creates a
> row with 60,000 columns and repeatedly updates it. Each column has a 18
> byte qualifier and a 50 byte value. In our tests, when we ran the
> program, we usually never got beyond 15 updates before it would flush
> for a really long time. The rows that are being updated are about 4MB
> each (minues any hbase metadata).
> It doesn't seem like it's caused by GC. I turned on gc logging, and
> didn't see any long pauses. This is the gc log during the flush.
> http://pastebin.com/vJKKXDx5
> This is the regionserver log with debug on during the same flush
> http://pastebin.com/Fh5213mg
> This is the test program we wrote.
> http://pastebin.com/aZ0k5tx2
> You should be able to just compile it, and run it against a running
> HBase cluster.
> $ java TestTable
> Carlos

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira