You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "Stian Soiland-Reyes (JIRA)" <ji...@apache.org> on 2015/03/19 05:14:39 UTC

[jira] [Comment Edited] (JENA-901) Make the cache of LPBRuleEngine bounded to avoid out-of-memory

    [ https://issues.apache.org/jira/browse/JENA-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368473#comment-14368473 ] 

Stian Soiland-Reyes edited comment on JENA-901 at 3/19/15 4:14 AM:
-------------------------------------------------------------------

You can call manually .reset() on the OntModel to clear the cache

{code}
    /**
     * Reset any internal caches. Some systems, such as the tabled backchainer, 
     * retain information after each query. A reset will wipe this information preventing
     * unbounded memory use at the expense of more expensive future queries. A reset
     * does not cause the raw data to be reconsulted and so is less expensive than a rebind.
     */
    public void reset();
{code}

But that does not really fix this bug - which is to limit the cache memory usage. 

I looked at using SoftReferences here - but it gets tricky as you would still have a Map with the stale TriplePatterns (which would contain all those millions of URIs). You can't combine this with a WeakHashMap - as the TriplePatterns are compared on equality.  Doing anything clever with clean-up of stale soft references on insertion could potentially become quite heavy, always iterating through the whole list. A ReferenceQueue combined with a Map gets tricky, as you would also need to keep the TriplePattern key somewhere in order to remove the whole Entry from the map (or do a slow iterate-purge at that point). Guava's caches would be more mature for this kind of logic - see https://code.google.com/p/guava-libraries/wiki/CachesExplained 

So I started with a fix using a BoundedMap with a maximum setting.

https://github.com/apache/jena/compare/master...stain:JENA-901-LPDRuleEngine-bound-cache?expand=1


The cache capacity can be configured by setting the system property `jena.rulesys.lp.max_cached_tabled_goals`

It is tricky to pick the right number - but we can find the lower bounds. 

I used the [sizeof](https://github.com/dweiss/java-sizeof) library to see how big the tabledGoals is after putting in 1024 entries from a very simple query- and found the memory footprint of LPBRuleEngine to increasing with about 1500 bytes per each iteration of this loop:

{code}
	        for (int i=0; i<MAX*4096; i++) {
	        	Node test = NodeFactory.createURI("test" + i);
	        	infgraph.find(test, ty, C2).close();
	        }
{code}

.. even if it is calling .close().  So more investigation needed for this memory leak. :(

To the profiler!


was (Author: soilandreyes):
You can call manually .reset() on the OntModel to clear the cache

    /**
     * Reset any internal caches. Some systems, such as the tabled backchainer, 
     * retain information after each query. A reset will wipe this information preventing
     * unbounded memory use at the expense of more expensive future queries. A reset
     * does not cause the raw data to be reconsulted and so is less expensive than a rebind.
     */
    public void reset();

But that does not really fix this bug - which is to limit the cache memory usage. 

I looked at using SoftReferences here - but it gets tricky as you would still have a Map with the stale TriplePatterns (which would contain all those millions of URIs). You can't combine this with a WeakHashMap - as the TriplePatterns are compared on equality.  Doing anything clever with clean-up of stale soft references on insertion could potentially become quite heavy, always iterating through the whole list. A ReferenceQueue combined with a Map gets tricky, as you would also need to keep the TriplePattern key somewhere in order to remove the whole Entry from the map (or do a slow iterate-purge at that point). Guava's caches would be more mature for this kind of logic - see https://code.google.com/p/guava-libraries/wiki/CachesExplained 

So I started with a fix using a BoundedMap with a maximum setting.

https://github.com/apache/jena/compare/master...stain:JENA-901-LPDRuleEngine-bound-cache?expand=1


The cache capacity can be configured by setting the system property `jena.rulesys.lp.max_cached_tabled_goals`

It is tricky to pick the right number - but we can find the lower bounds. 

I used the [sizeof](https://github.com/dweiss/java-sizeof) library to see how big the tabledGoals is after putting in 1024 entries from a very simple query- and found the memory footprint of LPBRuleEngine to increasing with about 1500 bytes per each iteration of this loop:

	        for (int i=0; i<MAX*4096; i++) {
	        	Node test = NodeFactory.createURI("test" + i);
	        	infgraph.find(test, ty, C2).close();
	        }

.. even if it is calling .close().  So more investigation needed for this memory leak. :(

To the profiler!

> Make the cache of LPBRuleEngine bounded to avoid out-of-memory
> --------------------------------------------------------------
>
>                 Key: JENA-901
>                 URL: https://issues.apache.org/jira/browse/JENA-901
>             Project: Apache Jena
>          Issue Type: Improvement
>          Components: Reasoners
>    Affects Versions: Jena 2.12.1
>            Reporter: Jan De Beer
>
> The class "com.hp.hpl.jena.reasoner.rulesys.impl.LPBRuleEngine" uses an in-memory cache named "tabledGoals", which has no limit as to the size/number of entries stored.
>     /** Table mapping tabled goals to generators for those goals.
>      *  This is here so that partial goal state can be shared across multiple queries. */
>     protected HashMap<TriplePattern, Generator> tabledGoals = new HashMap<>();
> We have experienced out-of-memory issues because of the cache being filled with millions of entries in just a few days under normal query usage conditions and a heap memory set to 3GB.
> In our setup, we have a dataset containing multiple graphs, some of them are actual data graphs (backed by TDB), and then there are two which are ontology models using a "TransitiveReasoner" and an "OWLMicroFBRuleReasoner", respectively. A typical query may run over all the graphs in the dataset, including the ontology ones (see below for a query template). Eventhough the ontology graphs would not yield any additional results for data queries (which is fine), the above mentioned cache would still fill up with new entries.
> SELECT ?p ?o
> WHERE {
>   GRAPH ?g {
>     <some resource of interest> ?p ?o .
>   }
> }
> As there is no upper bound to the cache, soon or later all available heap memory will be consumed by the cache, giving rise to an out-of-memory criticial error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)