[jira] [Created] (SVN-4796) SVN leaks exponentially significant amount of memory on commit

["BINETIX (JIRA)" <ji...@apache.org>]

BINETIX created SVN-4796:
----------------------------

             Summary: SVN leaks exponentially significant amount of memory on commit
                 Key: SVN-4796
                 URL: https://issues.apache.org/jira/browse/SVN-4796
             Project: Subversion
          Issue Type: Bug
          Components: libsvn_delta, libsvn_fs, libsvn_fs_fs, libsvn_fs_x, libsvn_repos, libsvn_subr
    Affects Versions: 1.10.2, 1.11.0
         Environment: Test environment:
 * Linux (Ubuntu 18.04.1 LTS)
 * 8 & 16GB RAM dev environments
 * GCC/G++ 7.3.0
 * Apache Portable Runtime Library, version 1.6.3-2 amd64
 * Repositories are created on EXT4 or XFS
 * Repositories are created on SSD
 * SVN repositories` type is FSFS
 * SVN repositories` file system version is the latest for the respective API
            Reporter: BINETIX


This is a clearly reproducible problem on SVN versions 1.10.2 and the latest 1.11.0.

We found two similar issues SVN-1964, SVN-3411 closed already, but obviously beyond our case. So, these are just mentioned for reference.

*Description*

We're going to use the SVN API in order to create and then commit multiple transactions with multiple files. In order to simplify the scenario, within the code below, we will apply only 3 actions per transaction:
 # Create a transaction
 # Add only one directory or one empty file (file with content makes no difference)
 # Commit the transaction

Now let`s loop many transactions like this one in sequence (no concurrent usage).

After each commit SVN consumes exponentially amount of memory which is never released afterwords, as follows (Number of transaction - Application's memory usage):

0 - 380 kb (initial state when the application is started with 0 transactions)

1`000 - 20 Mb

2`000 - 66 Mb

3`000 - 120 Mb

4`000 - 210 Mb

5`000 - 320 Mb

6`000 - 455 Mb

7`000 - 607 Mb

8`000 - 808 Mb

9`000 - 1.1 Gb

10`000 - 1.4 Gb

--

*Observations*

Few curious facts observed during the tests:
 * If we delete the added file before committing, the transaction remains empty, then there is no memory leakage;
 * If we add not only one file or dir but 40`000 - 48`000 entities in one transaction then the files are being created VERY slowly, and the memory consumption is doubled or even tippled compared to the table above;
 * If we create and commit 10`000 entities (in our example) in the root directory (it is important to do that in the root), and we stop the application, then on the next aplication start with the same repository there are three cases:
 ** if we add more entities in the root again, then the memory leakage is increased with the same proportions, as described above, BUT,
 ** if we add more entities in subdirectory, then the memory allocation is stopped by 20 Mb until a threshold of about 9`000 files is reached. Above this relative threshold, if we add more entities in the same subdirectory then the memory leakage is increased again, AND FINALLY,
 ** if we add more entities in any other subdirectory below the described relative threshold, then there is leakage observed. In other words, we create (relatively speaking) 10`000 files in the root, then we can add up to 9`000 in each subdirectory without memory leakage;
 * On Transaction destroy there is no memory leakage;

Despite these observations and strange "workarounds", obviously we CAN NOT use Subversion in productions conditions. So we hope you will find a way to fix these issues.

--

*Sample code for reproduction*

{{_#define_ _MAX_LOOP 10000_}}

{{_svn_error_t* test_func(const char* _szRepository_path__, apr_pool_t*     pool)_}}
{{ _{_}}
{{     _svn_repos_t*    repos;_}}
{{     _svn_fs_t*       fs;_}}

{{     _SVN_ERR(svn_repos_open3(&repos, _szRepository_path, NULL, pool, pool));_}}


{{     _fs = svn_repos_fs(repos);_}}

{{     _apr_pool_t* subpool = svn_pool_create(pool);_}}

{{     _for (int i = 0; i < MAX_LOOP; i++) {_}}
{{         _svn_fs_txn_t*   txn;_}}
{{         _svn_fs_root_t*  txn_root;_}}
{{         _svn_revnum_t    youngest;_}}
{{         _const char*     txn_id = NULL;_}}

{{         _svn_pool_clear(subpool);_}}
{{         __         }}
{{         _SVN_ERR(svn_fs_youngest_rev(&youngest, fs, subpool));_}}

{{         _SVN_ERR(svn_repos_fs_begin_txn_for_commit (&txn_}}
{{                                              _, repos_}}
{{                                              _, youngest_}}
{{                                              _, "user_name"_}}
{{                                              _, "comment default"_}}
{{                                              _, subpool));_}}

{{        __        }}

{{        _//---- Create unique file name and add to "/" in repository_}}
{{         _SVN_ERR(svn_fs_txn_root(&txn_root, txn, subpool));_}}
{{         __         }}

{{        _const char* pszFile_name = create_unique_file_name (subpool);_}}
{{         _SVN_ERR(svn_fs_make_file(txn_root, pszFile_name, subpool));_}}

{{         _if ((i % 100) == 0)_}}
{{             _printf ("added: %d\n", i+1);_}}

{{         _const char* conflict = NULL;_}}
{{         _svn_revnum_t new_rev = SVN_INVALID_REVNUM;_}}
{{    __    }}
{{         _SVN_ERR(svn_fs_commit_txn(&conflict, &new_rev, txn, subpool));_}}{{        __        }}

{{        _if (!SVN_IS_VALID_REVNUM(new_rev))_}}{{  __  }}

{{          _return svn_error_create(SVN_ERR_FS_GENERAL, NULL,_ _"Commit failed...");_}}{{    __    }}

{{    _}_}}

{{     _svn_pool_destroy(subpool);_}}

{{     _return SVN_NO_ERROR;_}}
{{ _}_}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)