[jira] [Updated] (HBASE-10925) Do not OOME, throw RowTooBigException instead

["Mikhail Antonov (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mikhail Antonov updated HBASE-10925:
------------------------------------

    Attachment: HBASE-10925.patch

First draft version of patch, to review general logic.

I actually found 2 scenarios of OOME, running tests with -Xmx128.

 - one, really artificial, but still: when row contains the few huge cells (like 10Mb each), and they are put and without compaction, than upon read the all store file scanners get sought after the key being looked up, and HRS fails with OOME even before actual scanner.next() is called.

Like that:

java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1487)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:413)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:846)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:155)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:176)
	at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1751)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:3691)
	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1876)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1862)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1839)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4699)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4674)
  
As a fix for that, I put the check in the code which does the initial seek in the scanner, just to illustrate. Not that it's valid usecase really?


 - second usecase, if when record just have a lot of columns, store files are compacted, but they scanners just attempt to load too much data. The fix I made is to calculate the total size of Cell elements which are getting appended to the result list.

Error for second case (without fix) looks like this:

java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1431)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:413)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:680)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:800)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:139)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:107)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:516)
	at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:139)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3823)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3903)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3771)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3753)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3740)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4700)
	at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4674)
	at 

> Do not OOME, throw RowTooBigException instead
> ---------------------------------------------
>
>                 Key: HBASE-10925
>                 URL: https://issues.apache.org/jira/browse/HBASE-10925
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Mikhail Antonov
>         Attachments: HBASE-10925.patch
>
>
> If 10M columns in a row, throw a RowTooBigException rather than OOME when Get'ing or Scanning w/o in-row scan flag set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)