Global Cache for TDB

["Dr. André Lanka" <ma...@dr-lanka.de>]

Hello Jena developers,

I know, you are working hard for the next Jena release. I just wanted to
let you know that I submitted a patch two weeks ago (JENA-524) It's
about a global cache that could be very useful if one's using many TDB
stores in direct file mode on a single machine. At least it's very
helpful for us and allows us to host more than five thousand tdb stores
(containing more than a billion triples) on a classical consumer PC. The
two main advantages for us are that
1. we can have concurrent write access to all the stores (what named
graphs don't provide).
2. we have a relatively small read cache of approx. 500 MB with a hit
rate of almost 99%.
3. Our JVM need approx. 2-3 GB of RAM for the whole system including TDB
access.

We will be very glad if you decide to integrate our approach to the
official release. So, if I can do something (provide further information
about the idea or something else) let me know. :)

Best wishes and thanks for the great piece of software you build
André

-- 
Dr. André Lanka  *  0178 / 134 44 47  *  http://dr-lanka.de

Re: Global Cache for TDB

[Andy Seaborne <an...@apache.org>]

On 10/09/13 18:13, "Dr. André Lanka" wrote:
> Hello Jena developers,
>
> I know, you are working hard for the next Jena release. I just wanted to
> let you know that I submitted a patch two weeks ago (JENA-524) It's
> about a global cache that could be very useful if one's using many TDB
> stores in direct file mode on a single machine. At least it's very
> helpful for us and allows us to host more than five thousand tdb stores
> (containing more than a billion triples) on a classical consumer PC. The
> two main advantages for us are that
> 1. we can have concurrent write access to all the stores (what named
> graphs don't provide).
> 2. we have a relatively small read cache of approx. 500 MB with a hit
> rate of almost 99%.
> 3. Our JVM need approx. 2-3 GB of RAM for the whole system including TDB
> access.
>
> We will be very glad if you decide to integrate our approach to the
> official release. So, if I can do something (provide further information
> about the idea or something else) let me know. :)
>
> Best wishes and thanks for the great piece of software you build
> André
>

André - thank you for the contribution.

I hope interested user will take the patch, apply and build an 
experimental system then provide feedback.  It looks good and I would 
like to understand the impact on all users.  It may well make sense to 
switch to only having per-JVM global caches as well as proper 
configurability.

As we're approaching a release and this is a significant change, I 
suggest we release in the current state then, with some clear space, 
investigate integrating this patch.

	Andy