svn commit: r1233733 - /hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java

[ns...@apache.org]

Author: nspiegelberg
Date: Fri Jan 20 02:23:09 2012
New Revision: 1233733

URL: http://svn.apache.org/viewvc?rev=1233733&view=rev
Log:
[HBASE-3421] Fetch one KV at a time at compaction (v2 of the fix)

Summary:
Compactions have been failing in production, bringing the regionserver
down with an OOM. The hypothesis is that there is a row with lots of KVs and
they are brought into memory all at once. This fix makes a compaction fetch one
KV at a time. Compared to the open-source fix
(https://issues.apache.org/jira/browse/HBASE-3421) we are setting the limit to
1
instead of a configurable number because there is no real batching benefit
(everything is already happening on the server side).

This is a second version of the fix, reverting the last one that changed
flush instead of compaction.

Test Plan: Unit tests. Push to dark launch.

Reviewers: kannan

Reviewed By: kannan

CC: gqchen, hbase-eng@lists, kannan

Differential Revision: https://phabricator.fb.com/D390382

Revert Plan: OK

Modified:
    hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java

Modified: hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java?rev=1233733&r1=1233732&r2=1233733&view=diff
==============================================================================
--- hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java (original)
+++ hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java Fri Jan 20 02:23:09 2012
@@ -590,7 +590,7 @@ public class Store extends SchemaConfigu
           final List<KeyValue> kvs = new ArrayList<KeyValue>();
           boolean hasMore;
           do {
-            hasMore = scanner.next(kvs, 1);
+            hasMore = scanner.next(kvs);
             if (!kvs.isEmpty()) {
               for (KeyValue kv : kvs) {
                 // If we know that this KV is going to be included always, then let us
@@ -1276,7 +1276,7 @@ public class Store extends SchemaConfigu
         // since scanner.next() can return 'false' but still be delivering data,
         // we have to use a do/while loop.
         ArrayList<KeyValue> kvs = new ArrayList<KeyValue>();
-        while (scanner.next(kvs)) {
+        while (scanner.next(kvs, 1)) {
           if (writer == null && !kvs.isEmpty()) {
             writer = createWriterInTmp(maxKeyCount, false);
           }