svn commit: r1550103 - /db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java

[ka...@apache.org]

Author: kahatlen
Date: Wed Dec 11 10:57:34 2013
New Revision: 1550103

URL: http://svn.apache.org/r1550103
Log:
DERBY-5416: SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the
heap is full at call time and then gets mostly garbage collected later
on.

Improve the accuracy of the code that estimates the memory requirement
of the sort buffer. When it detects that the current memory usage is
lower than the initial memory usage, it now records the current usage
and uses that value instead of the initial memory usage in future
calculations.

This compensates to some degree, but not fully, for the skew in the
estimates due to garbage collection happening after the initial memory
usage. The memory requirement will not be as badly underestimated, and
the likelihood of OutOfMemoryErrors is reduced.

There is no regression test case for this bug, since the only
reliable, reproducible test case that we currently have, needs too
much time, disk space and memory to be included in the regression test
suite.

Modified:
    db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java

Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java
URL: http://svn.apache.org/viewvc/db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java?rev=1550103&r1=1550102&r2=1550103&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/MergeInserter.java Wed Dec 11 10:57:34 2013
@@ -62,9 +62,7 @@ final class MergeInserter implements Sor
 	Information about memory usage to dynamically tune the
 	in-memory sort buffer size.
 	*/
-	private long beginFreeMemory;
-	private long beginTotalMemory;
-	private long estimatedMemoryUsed;
+    private long beginMemoryUsage;
 	private boolean avoidMergeRun;		// try to avoid merge run if possible
     private int runSize;
     private int totalRunSize;
@@ -121,6 +119,8 @@ final class MergeInserter implements Sor
 
                 long currentFreeMemory = jvm.freeMemory();
                 long currentTotalMemory = jvm.totalMemory();
+                long currentMemoryUsage =
+                        currentTotalMemory - currentFreeMemory;
 
 				// before we create an external sort, which is expensive, see if
                 // we can use up more in-memory sort buffer
@@ -128,8 +128,8 @@ final class MergeInserter implements Sor
 				// beginning of the sort.  Not all of this memory is used by
 				// the sort and GC may have kicked in and release some memory.
 				// But it is a rough guess.
-        		estimatedMemoryUsed = (currentTotalMemory-currentFreeMemory) -
-		   			(beginTotalMemory-beginFreeMemory);
+                long estimatedMemoryUsed =
+                        currentMemoryUsage - beginMemoryUsage;
 
  				if (SanityManager.DEBUG)
                 {
@@ -148,6 +148,17 @@ final class MergeInserter implements Sor
                     }
                 }
 
+                if (estimatedMemoryUsed < 0) {
+                    // We use less memory now than before we started filling
+                    // the sort buffer, probably because gc has happened. This
+                    // means we don't have a good estimate for how much memory
+                    // the sort buffer has occupied. To compensate for that,
+                    // set the begin memory usage to the current memory usage,
+                    // so that we get a more correct (but probably still too
+                    // low) estimate the next time we get here. See DERBY-5416.
+                    beginMemoryUsage = currentMemoryUsage;
+                }
+
 				// we want to double the sort buffer size if that will result
 				// in the sort to use up no more than 1/2 of all the free
 				// memory (including the sort memory)
@@ -267,9 +278,7 @@ final class MergeInserter implements Sor
             }
         }
 
-		beginFreeMemory = jvm.freeMemory();
-		beginTotalMemory = jvm.totalMemory();
-		estimatedMemoryUsed = 0;
+        beginMemoryUsage = jvm.totalMemory() - jvm.freeMemory();
 		avoidMergeRun = true;		// not an external sort
         stat_sortType = "internal";
         stat_numMergeRuns = 0;