You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Jean-Marc Vanel <je...@gmail.com> on 2020/09/02 16:03:17 UTC
Memory increase in TDB 1 incremental load
To load a big Turtle without stopping my (non Fuseki) server, I did an HTTP
client that sends the triples, about 300_000, by chunks of 10_000, talking
to a SPARQL 1.1 Graph Store HTTP Protocol
<http://www.w3.org/TR/2013/REC-sparql11-http-rdf-update-20130321> service.
The Java heap memory keeps growing until the final crash.
There were no interactive users at the time.
Following advice in
https://jena.apache.org/documentation/tdb/faqs.html#fuseki-tdb-memory-leak
, I tried sync() and dataset.close() on a newly created Dataset, but no
improvement.
Here is a sketch what my code does:
val dataset = TDBFactory.createDataset("TDBtest")
dataset.begin(ReadWrite.WRITE)
val dsg = dataset.asDatasetGraph
dsg.add(
NodeFactory.createURI("urn:gr" + i),
NodeFactory.createURI("urn:s"),
NodeFactory.createURI("urn:p"),
NodeFactory.createURI("urn:o"))
TDB.sync(dataset)
dataset.commit()
dataset.close()
Note that there is a singleton Dataset that is used for the interactive
HTML usage.
I tried this
if( ! localDataset . isInTransaction)
TDBFactory.release(localDataset)
but it gives:
org.apache.jena.sparql.JenaTransactionException: Already closed
at
org.apache.jena.tdb.transaction.DatasetGraphTransaction.checkNotClosed(DatasetGraphTransaction.java:337)
at
org.apache.jena.tdb.transaction.DatasetGraphTransaction.isInTransaction(DatasetGraphTransaction.java:233)
at
org.apache.jena.sparql.core.DatasetImpl.isInTransaction(DatasetImpl.java:146)
Would it be a use case for moving to TDB2?
I should look at what tdbloader does ...
Jean-Marc Vanel
+33 (0)6 89 16 29 52
Réseau social http://semantic-forms.cc:1952/
Re: Memory increase in TDB 1 incremental load
Posted by Andy Seaborne <an...@apache.org>.
On 04/09/2020 09:52, Jean-Marc Vanel wrote:
> So I could FIX my problem of OOM exception during incremental TBD 1 load.
> It boiled down to
>
> 1. QueueBatchSize = 0 // same as Fuseki,
> 2. raise memory from 900M to 1100M.
> 3. avoid unnecessary things like sync(), dataset.close() ; use only one
> of the patterns for transaction
>
> The memory kept around there figures:
> Used Memory:534.842
> Free Memory:565.158
> Total Memory:1100.0
> Max Memory:1100.0
>
> Maybe there is a tendency to raise on the average; I'll plot these data ...
> Understandably, the delay for a HTML page during the load raised to 10 to
> 20 seconds, but this is tolerable as long as this does not kill the server.
> My catching of OOM exceptions is not perfect :( .
>
> NOTES:
> I'm not sure if Jena TDB loading transaction does all the flushing and
> memory cleaning in the same thread; otherwise a delay between requests is
> necessary.
write-back to the main database and releasing the journal use can happen
on any request thread. E.g. when an reader R exits, the system may now
be able to do that work so an R can still cause base-database update work.
The only time a delay might be needed is flooding a server with
multiple/parallel write requests which might lockout the write-back. But
that isn't your use case of one sender - many sequential writes.
> I couldn't find how to get access to an instance of TransactionManager to
> call these methods : getCountActiveReaders() ; getCountActiveWriters ;
> getCountActiveWriters() ; getQueueLength .
TDBInternal
NB: "internal".
Their exact meaning and semantics is what the system does in the version
you are running.
Andy
>
> Le jeu. 3 sept. 2020 à 18:49, Andy Seaborne <an...@apache.org> a écrit :
>
>> If necessary, you can reduce the footprint of the node cache. See
>> StoreParams. But first, determine the root cause.
>> https://jena.apache.org/documentation/tdb/store-parameters.html
>> tdb.node2nodeid_cache_size
>> tdb.nodeid2node_cache_size
>> (BTW they are larger in TDB2)
>>>> The node table cache also takes
>>>> space (which stabilizes after a while - it's LRU).
>>
>
> I keep this under my elbow ..
>
> Jean-Marc Vanel
> <http://semantic-forms.cc:9112/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
> +33
> (0)6 89 16 29 52
>
>>>>>>> Réseau social http://semantic-forms.cc:1952/
>>
>>
>
Re: Memory increase in TDB 1 incremental load
Posted by Jean-Marc Vanel <je...@gmail.com>.
So I could FIX my problem of OOM exception during incremental TBD 1 load.
It boiled down to
1. QueueBatchSize = 0 // same as Fuseki,
2. raise memory from 900M to 1100M.
3. avoid unnecessary things like sync(), dataset.close() ; use only one
of the patterns for transaction
The memory kept around there figures:
Used Memory:534.842
Free Memory:565.158
Total Memory:1100.0
Max Memory:1100.0
Maybe there is a tendency to raise on the average; I'll plot these data ...
Understandably, the delay for a HTML page during the load raised to 10 to
20 seconds, but this is tolerable as long as this does not kill the server.
My catching of OOM exceptions is not perfect :( .
NOTES:
I'm not sure if Jena TDB loading transaction does all the flushing and
memory cleaning in the same thread; otherwise a delay between requests is
necessary.
I couldn't find how to get access to an instance of TransactionManager to
call these methods : getCountActiveReaders() ; getCountActiveWriters ;
getCountActiveWriters() ; getQueueLength .
Le jeu. 3 sept. 2020 à 18:49, Andy Seaborne <an...@apache.org> a écrit :
> If necessary, you can reduce the footprint of the node cache. See
> StoreParams. But first, determine the root cause.
> https://jena.apache.org/documentation/tdb/store-parameters.html
> tdb.node2nodeid_cache_size
> tdb.nodeid2node_cache_size
> (BTW they are larger in TDB2)
> > > The node table cache also takes
> >> space (which stabilizes after a while - it's LRU).
>
I keep this under my elbow ..
Jean-Marc Vanel
<http://semantic-forms.cc:9112/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
+33
(0)6 89 16 29 52
> >>>>> Réseau social http://semantic-forms.cc:1952/
>
>
Re: Memory increase in TDB 1 incremental load
Posted by Andy Seaborne <an...@apache.org>.
On 03/09/2020 15:35, Jean-Marc Vanel wrote:
> Le jeu. 3 sept. 2020 à 14:35, Andy Seaborne <an...@apache.org> a écrit :
>
>>> Used Memory:794
>>> Free Memory:105
>>> loadAction: AFTER System.gc(): Free Memory: 137 Mb
>>> java.lang.OutOfMemoryError: Java heap space
>>> Used Memory:892
>>> Free Memory:7
>>
>
>
>> You'll get more useful figures if you run -Xms and -Xmx set to the same
>> number.
>>
> Indeed that's what man java recommends for a server. Only -Xmx was set.
> Should I care about -Xss and -Xmn ?
Not relevant here.
>> Runtime.getFreeMemory does not reflect the possibility of
>> growing the heap.
>>
> Then what could I use ?
max - total
https://stackoverflow.com/questions/3571203/what-are-runtime-getruntime-totalmemory-and-freememory
If -Xms is set the same as -Xmx then "free" should be rightish.
"freeMemory" is "current heap size - in-use" and "current heap size may
grow.
Andy
>
>
>> All I can see is that:
>> Some batch is much larger (in bytes) than others.
>> The total heap size isn't very big.
>
>
> I raise it to 1100M ; I only have a total of 4Gb at the moment .
The default size for Fuseki used to be 1.5G without problems.
If necessary, you can reduce the footprint of the node cache. See
StoreParams. But first, determine the root cause.
https://jena.apache.org/documentation/tdb/store-parameters.html
tdb.node2nodeid_cache_size
tdb.nodeid2node_cache_size
(BTW they are larger in TDB2)
>
> The node table cache also takes
>> space (which stabilizes after a while - it's LRU).
>>
>
> Are there some indicators in the API to print for a TDB dashboard ?
> Is there already a doc. article about tuning TDB and the JVM ?
>
> I'll definitely try TDB2
>>
>>>> Jean-Marc Vanel
>>>>> +33 (0)6 89 16 29 52
>>>>> Réseau social http://semantic-forms.cc:1952/
>>
>
Re: Memory increase in TDB 1 incremental load
Posted by Jean-Marc Vanel <je...@gmail.com>.
Le jeu. 3 sept. 2020 à 14:35, Andy Seaborne <an...@apache.org> a écrit :
> > Used Memory:794
> > Free Memory:105
> > loadAction: AFTER System.gc(): Free Memory: 137 Mb
> > java.lang.OutOfMemoryError: Java heap space
> > Used Memory:892
> > Free Memory:7
>
> You'll get more useful figures if you run -Xms and -Xmx set to the same
> number.
>
Indeed that's what man java recommends for a server. Only -Xmx was set.
Should I care about -Xss and -Xmn ?
> Runtime.getFreeMemory does not reflect the possibility of
> growing the heap.
>
Then what could I use ?
> All I can see is that:
> Some batch is much larger (in bytes) than others.
> The total heap size isn't very big.
I raise it to 1100M ; I only have a total of 4Gb at the moment .
The node table cache also takes
> space (which stabilizes after a while - it's LRU).
>
Are there some indicators in the API to print for a TDB dashboard ?
Is there already a doc. article about tuning TDB and the JVM ?
I'll definitely try TDB2
>
> >> Jean-Marc Vanel
> >>> +33 (0)6 89 16 29 52
> >>> Réseau social http://semantic-forms.cc:1952/
>
Re: Memory increase in TDB 1 incremental load
Posted by Andy Seaborne <an...@apache.org>.
On 03/09/2020 10:14, Jean-Marc Vanel wrote:
> Le mer. 2 sept. 2020 à 23:39, Andy Seaborne <an...@apache.org> a écrit :
>
>> On 02/09/2020 17:03, Jean-Marc Vanel wrote:
>>> To load a big Turtle without stopping my (non Fuseki) server, I did an
>> HTTP
>>> client that sends the triples, about 3_000_000 (300_000 subjects), by
>> chunks of 10_000, talking
>>> to a SPARQL 1.1 Graph Store HTTP Protocol
>>> The Java heap memory keeps growing until the final crash.
>>
>> After how many chunks?
>>
> 12
> Here is a grep from the logs:
> Used Memory:36
> Free Memory:56
> loadAction: AFTER System.gc(): Free Memory: 64 Mb
> Used Memory:59
> Free Memory:297
> loadAction: AFTER System.gc(): Free Memory: 76 Mb
> Used Memory:218
> Free Memory:259
> loadAction: AFTER System.gc(): Free Memory: 303 Mb
> Used Memory:405
> Free Memory:153
> loadAction: AFTER System.gc(): Free Memory: 240 Mb
> Used Memory:555
> Free Memory:137
> loadAction: AFTER System.gc(): Free Memory: 318 Mb
> Used Memory:655
> Free Memory:131
> loadAction: AFTER System.gc(): Free Memory: 291 Mb
> Used Memory:366
> Free Memory:533
> loadAction: AFTER System.gc(): Free Memory: 76 Mb
> Used Memory:227
> Free Memory:151
> loadAction: AFTER System.gc(): Free Memory: 206 Mb
> Used Memory:463
> Free Memory:214
> loadAction: AFTER System.gc(): Free Memory: 359 Mb
> Used Memory:576
> Free Memory:101
> loadAction: AFTER System.gc(): Free Memory: 319 Mb
> Used Memory:702
> Free Memory:105
> loadAction: AFTER System.gc(): Free Memory: 287 Mb
> Used Memory:794
> Free Memory:105
> loadAction: AFTER System.gc(): Free Memory: 137 Mb
> java.lang.OutOfMemoryError: Java heap space
> Used Memory:892
> Free Memory:7
You'll get more useful figures if you run -Xms and -Xmx set to the same
number. Runtime.getFreeMemory does not relflect the possibility of
growing the heap.
All I can see is that:
Some batch is much larger (in bytes) than others.
The total heap size isn't very big. The node table cache also takes
space (which stabilizes after a while - it's LRU).
> Used Memory:893
> Free Memory:6
> Uncaught error from thread [application-scheduler-1]: Java heap space,
> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for
> ActorSystem[application]
>
> TDB1 batches commits and only writes back every 10 commits (if it can -
>> no transaction bloking the momentary sync that is needed).
>> Fuseki switches this off: TransactionManager.QueueBatchSize = 0;
>>
>
> I forgot that I already had set *QueueBatchSize = 5* after similar
> problems, on your advice :) . I will now go down to 2.
Try 0.
Try TDB2
> There is also the setting JournalThresholdSize, but the javadoc does
> specify the unit: triples, bytes ?
> Also, the difference between MaxQueueThreshold and QueueBatchSize is not
> clear enough, unless one looks at the issues mentioned.
> I also saw these useful indicators :
> getCountActiveReaders()
> getCountActiveWriters()
> getQueueLength()
> I should propose enhancements of the documentation after having solved my
> problems.
You do not want to alter MaxQueueThreshold or (especially not)
JournalThresholdSize.
These are internal mechanisms.
> But 10 chunks of 10K triples is small so either the heap size if too
>> small or something is stopping every 10th write back.
>>
>
> I don't see what it could be, but I hope that the above indicators will
> give a hint.
>
> ...
>>> TDB.sync(dataset)
>> Pointless. It is a transaction.
>> Don't call close(). Call dataset.end
>>
> I applied these advices...
>
> or better use Txn:
>> Txn.executeWrite(dataset, ()->{
>>
>
> I use Banana-RDF <https://github.com/banana-rdf/banana-rdf>, a Scala
> library that has a similar feature:
> def rw[T](dataset: Dataset, body: => T): Try[T] = Try {
> dataset.begin(ReadWrite.WRITE)
> try {
> val result = body
> dataset.commit()
> result
> } finally dataset.end()
> }
See the Txn code which does some other stuff as well.
> A good thing is that a Scala Lambda function can be passed to a Java
> library, and vice versa.
>
>> Jean-Marc Vanel
>>> +33 (0)6 89 16 29 52
>>> Réseau social http://semantic-forms.cc:1952/
>>
>
Re: Memory increase in TDB 1 incremental load
Posted by Jean-Marc Vanel <je...@gmail.com>.
Le mer. 2 sept. 2020 à 23:39, Andy Seaborne <an...@apache.org> a écrit :
> On 02/09/2020 17:03, Jean-Marc Vanel wrote:
> > To load a big Turtle without stopping my (non Fuseki) server, I did an
> HTTP
> > client that sends the triples, about 3_000_000 (300_000 subjects), by
> chunks of 10_000, talking
> > to a SPARQL 1.1 Graph Store HTTP Protocol
> > The Java heap memory keeps growing until the final crash.
>
> After how many chunks?
>
12
Here is a grep from the logs:
Used Memory:36
Free Memory:56
loadAction: AFTER System.gc(): Free Memory: 64 Mb
Used Memory:59
Free Memory:297
loadAction: AFTER System.gc(): Free Memory: 76 Mb
Used Memory:218
Free Memory:259
loadAction: AFTER System.gc(): Free Memory: 303 Mb
Used Memory:405
Free Memory:153
loadAction: AFTER System.gc(): Free Memory: 240 Mb
Used Memory:555
Free Memory:137
loadAction: AFTER System.gc(): Free Memory: 318 Mb
Used Memory:655
Free Memory:131
loadAction: AFTER System.gc(): Free Memory: 291 Mb
Used Memory:366
Free Memory:533
loadAction: AFTER System.gc(): Free Memory: 76 Mb
Used Memory:227
Free Memory:151
loadAction: AFTER System.gc(): Free Memory: 206 Mb
Used Memory:463
Free Memory:214
loadAction: AFTER System.gc(): Free Memory: 359 Mb
Used Memory:576
Free Memory:101
loadAction: AFTER System.gc(): Free Memory: 319 Mb
Used Memory:702
Free Memory:105
loadAction: AFTER System.gc(): Free Memory: 287 Mb
Used Memory:794
Free Memory:105
loadAction: AFTER System.gc(): Free Memory: 137 Mb
java.lang.OutOfMemoryError: Java heap space
Used Memory:892
Free Memory:7
Used Memory:893
Free Memory:6
Uncaught error from thread [application-scheduler-1]: Java heap space,
shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for
ActorSystem[application]
TDB1 batches commits and only writes back every 10 commits (if it can -
> no transaction bloking the momentary sync that is needed).
> Fuseki switches this off: TransactionManager.QueueBatchSize = 0;
>
I forgot that I already had set *QueueBatchSize = 5* after similar
problems, on your advice :) . I will now go down to 2.
There is also the setting JournalThresholdSize, but the javadoc does
specify the unit: triples, bytes ?
Also, the difference between MaxQueueThreshold and QueueBatchSize is not
clear enough, unless one looks at the issues mentioned.
I also saw these useful indicators :
getCountActiveReaders()
getCountActiveWriters()
getQueueLength()
I should propose enhancements of the documentation after having solved my
problems.
But 10 chunks of 10K triples is small so either the heap size if too
> small or something is stopping every 10th write back.
>
I don't see what it could be, but I hope that the above indicators will
give a hint.
...
> > TDB.sync(dataset)
> Pointless. It is a transaction.
> Don't call close(). Call dataset.end
>
I applied these advices...
or better use Txn:
> Txn.executeWrite(dataset, ()->{
>
I use Banana-RDF <https://github.com/banana-rdf/banana-rdf>, a Scala
library that has a similar feature:
def rw[T](dataset: Dataset, body: => T): Try[T] = Try {
dataset.begin(ReadWrite.WRITE)
try {
val result = body
dataset.commit()
result
} finally dataset.end()
}
A good thing is that a Scala Lambda function can be passed to a Java
library, and vice versa.
> Jean-Marc Vanel
> > +33 (0)6 89 16 29 52
> > Réseau social http://semantic-forms.cc:1952/
>
Re: Memory increase in TDB 1 incremental load
Posted by Andy Seaborne <an...@apache.org>.
On 02/09/2020 17:03, Jean-Marc Vanel wrote:
> To load a big Turtle without stopping my (non Fuseki) server, I did an HTTP
> client that sends the triples, about 300_000, by chunks of 10_000, talking
> to a SPARQL 1.1 Graph Store HTTP Protocol
> <http://www.w3.org/TR/2013/REC-sparql11-http-rdf-update-20130321> service.
>
> The Java heap memory keeps growing until the final crash.
After how many chunks?
TDB1 batches commits and only writes back every 10 commits (if it can -
no transaction bloking the momentary sync that is needed).
Fuseki switches this off:
TransactionManager.QueueBatchSize = 0;
But 10 chunks of 10K triples is small so either the heap size if too
small or something is stopping every 10th write back.
TDB2 does not have limitations on transaction size.
> There were no interactive users at the time.
>
> Following advice in
> https://jena.apache.org/documentation/tdb/faqs.html#fuseki-tdb-memory-leak
> , I tried sync() and dataset.close() on a newly created Dataset, but no
> improvement.
> Here is a sketch what my code does:
> val dataset = TDBFactory.createDataset("TDBtest")
> dataset.begin(ReadWrite.WRITE)
> val dsg = dataset.asDatasetGraph
> dsg.add(
> NodeFactory.createURI("urn:gr" + i),
> NodeFactory.createURI("urn:s"),
> NodeFactory.createURI("urn:p"),
> NodeFactory.createURI("urn:o"))
> TDB.sync(dataset)
Pointless. It is a transaction.
> dataset.commit()
> dataset.close()
Don't call close().
Call dataset.end
or better use Txn:
Txn.executeWrite(dataset, ()->{
val dsg = dataset.asDatasetGraph
dsg.add(
NodeFactory.createURI("urn:gr" + i),
NodeFactory.createURI("urn:s"),
NodeFactory.createURI("urn:p"),
NodeFactory.createURI("urn:o"))
});
>
> Note that there is a singleton Dataset that is used for the interactive
> HTML usage.
> I tried this
> if( ! localDataset . isInTransaction)
> TDBFactory.release(localDataset)
>
> but it gives:
> org.apache.jena.sparql.JenaTransactionException: Already closed
You called close.
> at
> org.apache.jena.tdb.transaction.DatasetGraphTransaction.checkNotClosed(DatasetGraphTransaction.java:337)
> at
> org.apache.jena.tdb.transaction.DatasetGraphTransaction.isInTransaction(DatasetGraphTransaction.java:233)
> at
> org.apache.jena.sparql.core.DatasetImpl.isInTransaction(DatasetImpl.java:146)
>
> Would it be a use case for moving to TDB2?
Yes.
> I should look at what tdbloader does ...
Only has advantages if the dataset is empty.
>
> Jean-Marc Vanel
> +33 (0)6 89 16 29 52
> Réseau social http://semantic-forms.cc:1952/
>