svn commit: r430578 - /db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java

[mi...@apache.org]

Author: mikem
Date: Thu Aug 10 15:57:40 2006
New Revision: 430578

URL: http://svn.apache.org/viewvc?rev=430578&view=rev
Log:
DERBY-1661, submitted by Sunitha 

Cause:
--On linux on the machine where the problem was hit, the default number for open file descriptors allowed is 1024. 
-- The problem lies in the sort code. The create index step involves the external merge sort.  There are some knobs in the code for the maximum number of input runs  to be used at a time. This is given by DEFAULT_MAX_MERGE_RUN in ExternalSortFactory.java. The default value is 1024.
--If the number of merge runs exceeds the limit given by DEFAULT_MAX_MERGE_RUN (lets call it 'x'), we combine the x number of runs into one larger run.  This process involves opening the x number of tmp files which is required to do the merge.
In this particular issue, when opening the 974th merge run, the too many files open is hit on this machine. 

See code:
org.apache.derby.impl.store.access.sort.MergeScan.init()

// Open a scan on each merge run.
int scanindex = 0;
Enumeration e = mergeRuns.elements();
while (e.hasMoreElements())
{
    // get the container id
    long id = ((Long) e.nextElement()).longValue();

    Transaction rawTran = tran.getRawStoreXact();  // get raw transaction
    int segmentId = StreamContainerHandle.TEMPORARY_SEGMENT;
    openScans[scanindex++] = 
    rawTran.openStreamContainer(segmentId, id, hold);   // <---------  This call throws the FileNotFoundException.
}

Workaround:
Increase the open file descriptor limit on the OS to a higher number. This solves the problem without any changes to derby.But this requires the user to explicitly change this limit on the machine, not a good option.
E.g on linux, the command to change the open file descriptor limit to 2048, 
$ulimit -n 2048 

To see the new /all limits, the command is
$ulimit -a


Fix to the problem:
1)change the DEFAULT_MAX_MERGE_RUN to 512.


Modified:
    db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java

Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java
URL: http://svn.apache.org/viewvc/db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java?rev=430578&r1=430577&r2=430578&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/store/access/sort/ExternalSortFactory.java Thu Aug 10 15:57:40 2006
@@ -66,7 +66,8 @@
 
 	protected static final int DEFAULT_MEM_USE = 1024*1024; // aim for about 1Meg
 	// how many sort runs to combined into a larger sort run
-	protected static final int DEFAULT_MAX_MERGE_RUN = 1024; 
+    // (DERBY-1661)
+	protected static final int DEFAULT_MAX_MERGE_RUN = 512; 
 
 	// sizeof Node + reference to Node + 12 bytes tax
 	private static final int SORT_ROW_OVERHEAD = 8*4+12;