svn commit: r1669955 - /subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c

[st...@apache.org]

Author: stefan2
Date: Sun Mar 29 19:21:05 2015
New Revision: 1669955

URL: http://svn.apache.org/r1669955
Log:
On the 1.8.x-memory-fragmentation branch:
Prevent excessive memory fragmentation in FSFS when activating fulltext
caching in Apache with MaxFreeMem set to 0 (unbounded, the default in 2.2).
Inspired by r1591005 on /trunk.

The default Apache 2.2 configuration does not limit the amount of unused
memory held by the worker thread's allocator.  Requests involving slowly
growing fulltext sizes (such as 'svn annotate') will often not fit into
an existing memory node but cause APR to allocate a new one.  This gives
O(n^2) worst-case memory consumption when fulltext caching is enabled.
Without fulltext caching, we don't allocate large, arbitrarily sized blocks
(arrays etc. already grow similarly to what we do here).

This fix is simple:  Pre-allocate the string buffer for larger fulltexts
as a power of 2 minus some room for overhead.  Because we only request few
different sizes, the respective memory nodes can be reused.  If N is the
size of the longest fulltext, the allocator will at most allocate 4N bytes
for fulltexts over its lifetime (assuming they never get released to the OS).

Since we only select the initial size of our auto-reallocating SVN string
buffer object, this patch will not affect the correctness of the remaining
code.

* subversion/libsvn_fs_fs/fs_fs.c
  (optimimal_allocation_size): New utility function. Taken from /trunk.
  (rep_read_get_baton): Use the new function to allocate large
                        buffers as blocks of 2^N bytes.

Modified:
    subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c

Modified: subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c
URL: http://svn.apache.org/viewvc/subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c?rev=1669955&r1=1669954&r2=1669955&view=diff
==============================================================================
--- subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c (original)
+++ subversion/branches/1.8.x-memory-fragmentation/subversion/libsvn_fs_fs/fs_fs.c Sun Mar 29 19:21:05 2015
@@ -4900,6 +4900,49 @@ build_rep_list(apr_array_header_t **list
     }
 }
 
+/* Determine the optimal size of a string buf that shall receive a
+ * (full-) text of NEEDED bytes.
+ *
+ * The critical point is that those buffers may be very large and
+ * can cause memory fragmentation.  We apply simple heuristics to
+ * make fragmentation less likely.
+ */
+static apr_size_t
+optimimal_allocation_size(apr_size_t needed)
+{
+  /* For all allocations, assume some overhead that is shared between
+   * OS memory managemnt, APR memory management and svn_stringbuf_t. */
+  const apr_size_t overhead = 0x400;
+  apr_size_t optimal;
+
+  /* If an allocation size if safe for other ephemeral buffers, it should
+   * be safe for ours. */
+  if (needed <= SVN__STREAM_CHUNK_SIZE)
+    return needed;
+
+  /* Paranoia edge case:
+   * Skip our heuristics if they created arithmetical overflow.
+   * Beware to make this test work for NEEDED = APR_SIZE_MAX as well! */
+  if (needed >= APR_SIZE_MAX / 2 - overhead)
+    return needed;
+
+  /* As per definition SVN__STREAM_CHUNK_SIZE is a power of two.
+   * Since we know NEEDED to be larger than that, use it as the
+   * starting point.
+   *
+   * Heuristics: Allocate a power-of-two number of bytes that fit
+   *             NEEDED plus some OVERHEAD.  The APR allocator
+   *             will round it up to the next full page size.
+   */
+  optimal = SVN__STREAM_CHUNK_SIZE;
+  while (optimal - overhead < needed)
+    optimal *= 2;
+
+  /* This is above or equal to NEEDED. */
+  return optimal - overhead;
+}
+
+
 
 /* Create a rep_read_baton structure for node revision NODEREV in
    filesystem FS and store it in *RB_P.  If FULLTEXT_CACHE_KEY is not
@@ -4935,7 +4978,7 @@ rep_read_get_baton(struct rep_read_baton
 
   if (SVN_IS_VALID_REVNUM(fulltext_cache_key.revision))
     b->current_fulltext = svn_stringbuf_create_ensure
-                            ((apr_size_t)b->len,
+                            (optimimal_allocation_size((apr_size_t)b->len),
                              b->filehandle_pool);
   else
     b->current_fulltext = NULL;