[jira] [Commented] (JENA-567) Improve Fuseki/TDB transaction memory usage

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/JENA-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804885#comment-13804885 ] 

ASF subversion and git services commented on JENA-567:
------------------------------------------------------

Commit 1535599 from [~sallen] in branch 'jena/trunk'
[ https://svn.apache.org/r1535599 ]

JENA-567 Add option to TDB for specifying where the temporary write blocks for the BlockMgrJournal come from.  Also added a memory mapped ByteBuffer allocator to work with the new BlockMgrJournal functionality.

> Improve Fuseki/TDB transaction memory usage
> -------------------------------------------
>
>                 Key: JENA-567
>                 URL: https://issues.apache.org/jira/browse/JENA-567
>             Project: Apache Jena
>          Issue Type: Improvement
>          Components: TDB
>            Reporter: Stephen Allen
>            Assignee: Stephen Allen
>
> TDB has to buffer in memory all of the modified blocks for a transaction before committing it.  This causes out of memory exceptions when attempting to add a large number of statements in a single transaction.
> An easy way to fix this would be to copy the write block contents into a memory mapped file instead of heap memory ^†^.  We can provide three user specified options for controlling the location of the temporary blocks:
> # JVM heap (default, and what we currently use)
> # Direct memory (process heap, but not in the JVM)
> # Memory mapped temporary file
> See this [Jena thread| http://markmail.org/thread/ckeevvhl2luevixw] for some additional discussion.
> ^†^ _The harder way would involve writing the old blocks to the journal, then writing the new blocks directly to the indexes, with a tombstone pointing to the old block in the journal so that readers could still retrieve the old version.  This however would seem to require a substantial refactor, as well as a change to the on-disk database format._



--
This message was sent by Atlassian JIRA
(v6.1#6144)