You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Jean-Marc Vanel <je...@gmail.com> on 2016/02/03 11:13:42 UTC
Memory leak in TDB using a single Dataset object
I have a repeating memory leak in TDB in my web application (
https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
).
It is caching RDF documents from internet, typically dbpedia ressources.
It is not the use case described in "Fuseki/TDB memory leak for concurrent
updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as the
journal is empty after crash .
A single Dataset object is used for the duration of the application, and I
suspect this is the root cause.
In the documentation,
https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
it is not clear which use pattern is preferred and the reason why.
You someone confirm that keeping a single Dataset object for the duration
of the application is bad ?
Re: Memory leak in TDB using a single Dataset object
Posted by Andy Seaborne <an...@apache.org>.
Ok - so it was the reduced heap size.
Good thing TDB uses out-of-heap space for indexes then! This space
(mapped files) flexes under OS control. It's the filing system cache
mapped into the user process.
The TDB union graph takes zero space - it's calculated.
What you could do is one transaction per HTTP request, assuming a HTTP
request is not long running. If you can identify what is a read request
and what is a write request,
Singe triple transactions (c.f. autocommit) is generally a bad idea for
use above small. The overhead costs mount up - fully compatible with
SQL here!
(TDB2 - any add/delete outside a transaction is a truely autocommited
transaction)
Thanks for the update, sharing experiences really is a great help,
Andy
On 07/02/16 09:16, Jean-Marc Vanel wrote:
> I applied Andy's hints:
> - TransactionManager.DEBUG = true
> - TransactionManager.QueueBatchSize = 5
>
> Before each Transaction, it prints the size of the List of Transaction's of
> in commitedAwaitingFlush. I also printed the size of file TDB/journal.jrnl
> . So I see both sizes grow until reaching QueueBatchSize and then both
> sizes returns to 0 .
>
> What happened in my case was that the transactions were rather big, several
> Mb. And the memory set for the application was rather small, 200Mb.
> So, with the default of QueueBatchSize = 10, depending on the inputs ( in
> the case causing OOM exception inputs are dbPedia URL's ), the available
> memory was sooner or later exceeded. This was certainly made worse by
> having one or two instances of the union graph around.
>
> I did not set QueueBatchSize to 0 for now, because this list of commits
> Awaiting Flush optimizes disk usage, but maybe I should set it to the
> maximum number of transactions during an HTTP request. Note that I have
> also transactions as small as a single triple (corresponding to user
> input).
>
> The instrumented TransactionManager.java is here:
> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/app/controllers/TransactionManager.java
>
> Thanks all for you help !
>
>
> 2016-02-04 12:50 GMT+01:00 Rob Vesse <rv...@dotnetrdf.org>:
>
>>
>> On 04/02/2016 10:41, "Jean-Marc Vanel" <je...@gmail.com> wrote:
>>
>>> The journal workspace you mention is on disk , isn't it ? My problem is
>>> not
>>> on disk at all.
>>
>> No
>>
>> The journal is both in-memory and on-disk as it is a Write ahead log for
>> failure recovery purposes, the disk portion preserves the data for failure
>> recovery while the in-memory portion provides data to the live instance.
>>
>> If there is a non-empty journal on disk then there is a corresponding
>> amount of memory within the JVM heap used to store the latest state of the
>> data for subsequent transactions while not overwriting the old state of
>> the data which ongoing transactions may still be accessing
>>
>> Rob
>>
>>
>>
>>
>>
>
>
Re: Memory leak in TDB using a single Dataset object
Posted by Jean-Marc Vanel <je...@gmail.com>.
I applied Andy's hints:
- TransactionManager.DEBUG = true
- TransactionManager.QueueBatchSize = 5
Before each Transaction, it prints the size of the List of Transaction's of
in commitedAwaitingFlush. I also printed the size of file TDB/journal.jrnl
. So I see both sizes grow until reaching QueueBatchSize and then both
sizes returns to 0 .
What happened in my case was that the transactions were rather big, several
Mb. And the memory set for the application was rather small, 200Mb.
So, with the default of QueueBatchSize = 10, depending on the inputs ( in
the case causing OOM exception inputs are dbPedia URL's ), the available
memory was sooner or later exceeded. This was certainly made worse by
having one or two instances of the union graph around.
I did not set QueueBatchSize to 0 for now, because this list of commits
Awaiting Flush optimizes disk usage, but maybe I should set it to the
maximum number of transactions during an HTTP request. Note that I have
also transactions as small as a single triple (corresponding to user
input).
The instrumented TransactionManager.java is here:
https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/app/controllers/TransactionManager.java
Thanks all for you help !
2016-02-04 12:50 GMT+01:00 Rob Vesse <rv...@dotnetrdf.org>:
>
> On 04/02/2016 10:41, "Jean-Marc Vanel" <je...@gmail.com> wrote:
>
> >The journal workspace you mention is on disk , isn't it ? My problem is
> >not
> >on disk at all.
>
> No
>
> The journal is both in-memory and on-disk as it is a Write ahead log for
> failure recovery purposes, the disk portion preserves the data for failure
> recovery while the in-memory portion provides data to the live instance.
>
> If there is a non-empty journal on disk then there is a corresponding
> amount of memory within the JVM heap used to store the latest state of the
> data for subsequent transactions while not overwriting the old state of
> the data which ongoing transactions may still be accessing
>
> Rob
>
>
>
>
>
--
Jean-Marc Vanel
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
http://deductions-software.com/
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Memory leak in TDB using a single Dataset object
Posted by Rob Vesse <rv...@dotnetrdf.org>.
On 04/02/2016 10:41, "Jean-Marc Vanel" <je...@gmail.com> wrote:
>The journal workspace you mention is on disk , isn't it ? My problem is
>not
>on disk at all.
No
The journal is both in-memory and on-disk as it is a Write ahead log for
failure recovery purposes, the disk portion preserves the data for failure
recovery while the in-memory portion provides data to the live instance.
If there is a non-empty journal on disk then there is a corresponding
amount of memory within the JVM heap used to store the latest state of the
data for subsequent transactions while not overwriting the old state of
the data which ongoing transactions may still be accessing
Rob
Re: Memory leak in TDB using a single Dataset object
Posted by Andy Seaborne <an...@apache.org>.
On 04/02/16 10:41, Jean-Marc Vanel wrote:
> Comment interleaved.
Ditto
>
> 2016-02-04 10:26 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>> On 04/02/16 08:15, Jean-Marc Vanel wrote:
>>
>>> Sorry for being vague.
>>> The RAM usage is growing, until crashing with an Out Of Memery exception.
>>>
>>
>> TDB uses a bounded amount of caching, though the journal workspace can
>> grow.
>>
>
> So, if TDB uses a bounded amount of caching *in memory* , there is nothing
> against using the same Dataset object as singleton.
> The journal workspace you mention is on disk , isn't it ? My problem is not
> on disk at all.
>
>
>> If there are lots of large literals, you'll need more heap.
>>
>
> The largest literals involved in the transactions are dbPedia abstracts; I
> would not call that "large" literals.
> Anyway the problem happens sooner or later, raising the memory does not
> help.
>
>
> The transaction system in TDB1 keeps up to 10 transaction buffered : you
>> can switch that off with:
>>
>> TransactionManager.QueueBatchSize = 0 ;
>>
>> then commits are flushed back to the main database as soon as possible.
>> That needs no readers about.
>>
>
> I'll try that too, and report .
>
>
>> If you have a reader that doesn't commit/end properly, the system can
>> never write to the main database.
>>
>
> That should not be possible, when I use a Scala construct that wraps the
> fragment of code inside a transaction and automatically calls commit or end.
Ptr?
And you don't pass out anything that is hanging onto the state inside
the transaction?
Are you using (indirectly) the Jena model API or using SPARQL?
>
> However, there some reads on the database that happen outside a transaction.
> This covers navigation by find() on a <urn:x-arq:UnionGraph> graph .
> During developments, everytime a runtime exception said "outside a
> transaction" , I fixed that. So the other cases were left outside a
> transaction; it is wrong ?
Yes.
(I though it complained once in transaction mode if a non-transaction
operation happened)
>
> ...
>
> It is a disk-backed dataset, not an TDB memory one?
>>
>
> Yes disk based.
Ok - it sounds like either the journal is locked or
You can try turning logging on for TransactionManager (level "DEBUG").
Andy
Re: Memory leak in TDB using a single Dataset object
Posted by Jean-Marc Vanel <je...@gmail.com>.
Comment interleaved.
2016-02-04 10:26 GMT+01:00 Andy Seaborne <an...@apache.org>:
> On 04/02/16 08:15, Jean-Marc Vanel wrote:
>
>> Sorry for being vague.
>> The RAM usage is growing, until crashing with an Out Of Memery exception.
>>
>
> TDB uses a bounded amount of caching, though the journal workspace can
> grow.
>
So, if TDB uses a bounded amount of caching *in memory* , there is nothing
against using the same Dataset object as singleton.
The journal workspace you mention is on disk , isn't it ? My problem is not
on disk at all.
> If there are lots of large literals, you'll need more heap.
>
The largest literals involved in the transactions are dbPedia abstracts; I
would not call that "large" literals.
Anyway the problem happens sooner or later, raising the memory does not
help.
The transaction system in TDB1 keeps up to 10 transaction buffered : you
> can switch that off with:
>
> TransactionManager.QueueBatchSize = 0 ;
>
> then commits are flushed back to the main database as soon as possible.
> That needs no readers about.
>
I'll try that too, and report .
> If you have a reader that doesn't commit/end properly, the system can
> never write to the main database.
>
That should not be possible, when I use a Scala construct that wraps the
fragment of code inside a transaction and automatically calls commit or end.
However, there some reads on the database that happen outside a transaction.
This covers navigation by find() on a <urn:x-arq:UnionGraph> graph .
During developments, everytime a runtime exception said "outside a
transaction" , I fixed that. So the other cases were left outside a
transaction; it is wrong ?
...
It is a disk-backed dataset, not an TDB memory one?
>
Yes disk based.
--
Jean-Marc Vanel
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
http://deductions-software.com/
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Memory leak in TDB using a single Dataset object
Posted by Andy Seaborne <an...@apache.org>.
On 04/02/16 08:15, Jean-Marc Vanel wrote:
> Sorry for being vague.
> The RAM usage is growing, until crashing with an Out Of Memery exception.
TDB uses a bounded amount of caching, though the journal workspace can grow.
If there are lots of large literals, you'll need more heap.
The transaction system in TDB1 keeps up to 10 transaction buffered : you
can switch that off with:
TransactionManager.QueueBatchSize = 0 ;
then commits are flushed back to the main database as soon as possible.
That needs no readers about.
If you have a reader that doesn't commit/end properly, the system can
never write to the main database.
If you have a system where there are always readers, it will grow but
you don't have that setup if the below is true:
> AFAIK transactions occur on the same thread started by the Play! framework
> and so do not overlap.
> About the "pattern of transactions" , I don't know what to answer. I there
> was a questionnaire I'd be glad to answer. Also I can instrument the code
> if there is some procedure.
>
> It is running with java version "1.8.0_65" , on Ubuntu 15.10 .
>
> The test I'm going to do is to call close() and refresh the Dataset when
> reaching 80% of the maximum memory .
It is a disk-backed dataset, not an TDB memory one?
Andy
>
>
> 2016-02-03 23:05 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>> Hi there -
>>
>> "memory leak" has possible several meaning, not sure which you you mean:
>>
>> * RAM usage is growing?
>> * Disk usage is growing?
>> * a specific file (the journal is growing)?
>>
>> What is the pattern of transactions? (how many, do they overlap?)
>>
>> Andy
>>
>>
>> On 03/02/16 17:47, Jean-Marc Vanel wrote:
>>
>>> I forgot to mention that I'm still using Jena 2.13.0 , due to Banana-RDF
>>> not having updated.
>>>
>>>
>>> 2016-02-03 18:43 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>>>
>>> I think that the second pattern "create a dataset object on the thread",
>>>> or rather in my case
>>>> "create a dataset object for one HTTP request"
>>>> is worth trying.
>>>>
>>>> And I want to know why the doc seems to prefer the first pattern.
>>>>
>>>> 2016-02-03 18:30 GMT+01:00 A. Soroka <aj...@virginia.edu>:
>>>>
>>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> In the documentation,
>>>>>>
>>>>>>
>>>>>>
>>>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>>
>>>>>>
>>>>>> it is not clear which use pattern is preferred and the reason why.
>>>>>>
>>>>>
>>>>> The first pattern shows a single dataset object being shared between
>>>>> threads, each of which operates a transaction against that object, and
>>>>> the
>>>>> second pattern is introduced with "or create a dataset object on the
>>>>> thread
>>>>> (the case above is preferred):”.
>>>>>
>>>>> As to why, I am not familiar enough with TDB to be sure, but there is a
>>>>> comment on the second pattern "Each thread has a separate dataset
>>>>> object;
>>>>> these safely share the same storage but have independent transactions.”
>>>>> that would seem to indicate that the second pattern is vulnerable to
>>>>> having
>>>>> conflicts between transactions opened against the two different dataset
>>>>> objects.
>>>>>
>>>>> ---
>>>>> A. Soroka
>>>>> The University of Virginia Library
>>>>>
>>>>>
>>>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> I have a repeating memory leak in TDB in my web application (
>>>>>>
>>>>>>
>>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
>>>>>
>>>>>> ).
>>>>>> It is caching RDF documents from internet, typically dbpedia
>>>>>> ressources.
>>>>>>
>>>>>> It is not the use case described in "Fuseki/TDB memory leak for
>>>>>>
>>>>> concurrent
>>>>>
>>>>>> updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as
>>>>>>
>>>>> the
>>>>>
>>>>>> journal is empty after crash .
>>>>>>
>>>>>> A single Dataset object is used for the duration of the application,
>>>>>>
>>>>> and I
>>>>>
>>>>>> suspect this is the root cause.
>>>>>> In the documentation,
>>>>>>
>>>>>>
>>>>>>
>>>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>>
>>>>>>
>>>>>> it is not clear which use pattern is preferred and the reason why.
>>>>>>
>>>>>> You someone confirm that keeping a single Dataset object for the
>>>>>>
>>>>> duration
>>>>>
>>>>>> of the application is bad ?
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Jean-Marc Vanel
>>>> Déductions SARL - Consulting, services, training,
>>>> Rule-based programming, Semantic Web
>>>> http://deductions-software.com/
>>>> +33 (0)6 89 16 29 52
>>>> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
Re: Memory leak in TDB using a single Dataset object
Posted by Jean-Marc Vanel <je...@gmail.com>.
Sorry for being vague.
The RAM usage is growing, until crashing with an Out Of Memery exception.
AFAIK transactions occur on the same thread started by the Play! framework
and so do not overlap.
About the "pattern of transactions" , I don't know what to answer. I there
was a questionnaire I'd be glad to answer. Also I can instrument the code
if there is some procedure.
It is running with java version "1.8.0_65" , on Ubuntu 15.10 .
The test I'm going to do is to call close() and refresh the Dataset when
reaching 80% of the maximum memory .
2016-02-03 23:05 GMT+01:00 Andy Seaborne <an...@apache.org>:
> Hi there -
>
> "memory leak" has possible several meaning, not sure which you you mean:
>
> * RAM usage is growing?
> * Disk usage is growing?
> * a specific file (the journal is growing)?
>
> What is the pattern of transactions? (how many, do they overlap?)
>
> Andy
>
>
> On 03/02/16 17:47, Jean-Marc Vanel wrote:
>
>> I forgot to mention that I'm still using Jena 2.13.0 , due to Banana-RDF
>> not having updated.
>>
>>
>> 2016-02-03 18:43 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>>
>> I think that the second pattern "create a dataset object on the thread",
>>> or rather in my case
>>> "create a dataset object for one HTTP request"
>>> is worth trying.
>>>
>>> And I want to know why the doc seems to prefer the first pattern.
>>>
>>> 2016-02-03 18:30 GMT+01:00 A. Soroka <aj...@virginia.edu>:
>>>
>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>>
>>>>> In the documentation,
>>>>>
>>>>>
>>>>>
>>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>
>>>>>
>>>>> it is not clear which use pattern is preferred and the reason why.
>>>>>
>>>>
>>>> The first pattern shows a single dataset object being shared between
>>>> threads, each of which operates a transaction against that object, and
>>>> the
>>>> second pattern is introduced with "or create a dataset object on the
>>>> thread
>>>> (the case above is preferred):”.
>>>>
>>>> As to why, I am not familiar enough with TDB to be sure, but there is a
>>>> comment on the second pattern "Each thread has a separate dataset
>>>> object;
>>>> these safely share the same storage but have independent transactions.”
>>>> that would seem to indicate that the second pattern is vulnerable to
>>>> having
>>>> conflicts between transactions opened against the two different dataset
>>>> objects.
>>>>
>>>> ---
>>>> A. Soroka
>>>> The University of Virginia Library
>>>>
>>>>
>>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>>
>>>>> I have a repeating memory leak in TDB in my web application (
>>>>>
>>>>>
>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
>>>>
>>>>> ).
>>>>> It is caching RDF documents from internet, typically dbpedia
>>>>> ressources.
>>>>>
>>>>> It is not the use case described in "Fuseki/TDB memory leak for
>>>>>
>>>> concurrent
>>>>
>>>>> updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as
>>>>>
>>>> the
>>>>
>>>>> journal is empty after crash .
>>>>>
>>>>> A single Dataset object is used for the duration of the application,
>>>>>
>>>> and I
>>>>
>>>>> suspect this is the root cause.
>>>>> In the documentation,
>>>>>
>>>>>
>>>>>
>>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>
>>>>>
>>>>> it is not clear which use pattern is preferred and the reason why.
>>>>>
>>>>> You someone confirm that keeping a single Dataset object for the
>>>>>
>>>> duration
>>>>
>>>>> of the application is bad ?
>>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Jean-Marc Vanel
>>> Déductions SARL - Consulting, services, training,
>>> Rule-based programming, Semantic Web
>>> http://deductions-software.com/
>>> +33 (0)6 89 16 29 52
>>> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>>>
>>>
>>
>>
>>
>
--
Jean-Marc Vanel
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
http://deductions-software.com/
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Memory leak in TDB using a single Dataset object
Posted by Andy Seaborne <an...@apache.org>.
Hi there -
"memory leak" has possible several meaning, not sure which you you mean:
* RAM usage is growing?
* Disk usage is growing?
* a specific file (the journal is growing)?
What is the pattern of transactions? (how many, do they overlap?)
Andy
On 03/02/16 17:47, Jean-Marc Vanel wrote:
> I forgot to mention that I'm still using Jena 2.13.0 , due to Banana-RDF
> not having updated.
>
>
> 2016-02-03 18:43 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>
>> I think that the second pattern "create a dataset object on the thread",
>> or rather in my case
>> "create a dataset object for one HTTP request"
>> is worth trying.
>>
>> And I want to know why the doc seems to prefer the first pattern.
>>
>> 2016-02-03 18:30 GMT+01:00 A. Soroka <aj...@virginia.edu>:
>>
>>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>> wrote:
>>>>
>>>> In the documentation,
>>>>
>>>>
>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>
>>>> it is not clear which use pattern is preferred and the reason why.
>>>
>>> The first pattern shows a single dataset object being shared between
>>> threads, each of which operates a transaction against that object, and the
>>> second pattern is introduced with "or create a dataset object on the thread
>>> (the case above is preferred):”.
>>>
>>> As to why, I am not familiar enough with TDB to be sure, but there is a
>>> comment on the second pattern "Each thread has a separate dataset object;
>>> these safely share the same storage but have independent transactions.”
>>> that would seem to indicate that the second pattern is vulnerable to having
>>> conflicts between transactions opened against the two different dataset
>>> objects.
>>>
>>> ---
>>> A. Soroka
>>> The University of Virginia Library
>>>
>>>
>>>> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>>> wrote:
>>>>
>>>> I have a repeating memory leak in TDB in my web application (
>>>>
>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
>>>> ).
>>>> It is caching RDF documents from internet, typically dbpedia ressources.
>>>>
>>>> It is not the use case described in "Fuseki/TDB memory leak for
>>> concurrent
>>>> updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as
>>> the
>>>> journal is empty after crash .
>>>>
>>>> A single Dataset object is used for the duration of the application,
>>> and I
>>>> suspect this is the root cause.
>>>> In the documentation,
>>>>
>>>>
>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>>>>
>>>> it is not clear which use pattern is preferred and the reason why.
>>>>
>>>> You someone confirm that keeping a single Dataset object for the
>>> duration
>>>> of the application is bad ?
>>>
>>>
>>
>>
>> --
>> Jean-Marc Vanel
>> Déductions SARL - Consulting, services, training,
>> Rule-based programming, Semantic Web
>> http://deductions-software.com/
>> +33 (0)6 89 16 29 52
>> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>>
>
>
>
Re: Memory leak in TDB using a single Dataset object
Posted by Jean-Marc Vanel <je...@gmail.com>.
I forgot to mention that I'm still using Jena 2.13.0 , due to Banana-RDF
not having updated.
2016-02-03 18:43 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
> I think that the second pattern "create a dataset object on the thread",
> or rather in my case
> "create a dataset object for one HTTP request"
> is worth trying.
>
> And I want to know why the doc seems to prefer the first pattern.
>
> 2016-02-03 18:30 GMT+01:00 A. Soroka <aj...@virginia.edu>:
>
>> > On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>> wrote:
>> >
>> > In the documentation,
>> >
>> >
>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>> >
>> > it is not clear which use pattern is preferred and the reason why.
>>
>> The first pattern shows a single dataset object being shared between
>> threads, each of which operates a transaction against that object, and the
>> second pattern is introduced with "or create a dataset object on the thread
>> (the case above is preferred):”.
>>
>> As to why, I am not familiar enough with TDB to be sure, but there is a
>> comment on the second pattern "Each thread has a separate dataset object;
>> these safely share the same storage but have independent transactions.”
>> that would seem to indicate that the second pattern is vulnerable to having
>> conflicts between transactions opened against the two different dataset
>> objects.
>>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>>
>> > On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
>> wrote:
>> >
>> > I have a repeating memory leak in TDB in my web application (
>> >
>> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
>> > ).
>> > It is caching RDF documents from internet, typically dbpedia ressources.
>> >
>> > It is not the use case described in "Fuseki/TDB memory leak for
>> concurrent
>> > updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as
>> the
>> > journal is empty after crash .
>> >
>> > A single Dataset object is used for the duration of the application,
>> and I
>> > suspect this is the root cause.
>> > In the documentation,
>> >
>> >
>> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>> >
>> > it is not clear which use pattern is preferred and the reason why.
>> >
>> > You someone confirm that keeping a single Dataset object for the
>> duration
>> > of the application is bad ?
>>
>>
>
>
> --
> Jean-Marc Vanel
> Déductions SARL - Consulting, services, training,
> Rule-based programming, Semantic Web
> http://deductions-software.com/
> +33 (0)6 89 16 29 52
> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>
--
Jean-Marc Vanel
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
http://deductions-software.com/
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Memory leak in TDB using a single Dataset object
Posted by Jean-Marc Vanel <je...@gmail.com>.
I think that the second pattern "create a dataset object on the thread", or
rather in my case
"create a dataset object for one HTTP request"
is worth trying.
And I want to know why the doc seems to prefer the first pattern.
2016-02-03 18:30 GMT+01:00 A. Soroka <aj...@virginia.edu>:
> > On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
> wrote:
> >
> > In the documentation,
> >
> >
> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
> >
> > it is not clear which use pattern is preferred and the reason why.
>
> The first pattern shows a single dataset object being shared between
> threads, each of which operates a transaction against that object, and the
> second pattern is introduced with "or create a dataset object on the thread
> (the case above is preferred):”.
>
> As to why, I am not familiar enough with TDB to be sure, but there is a
> comment on the second pattern "Each thread has a separate dataset object;
> these safely share the same storage but have independent transactions.”
> that would seem to indicate that the second pattern is vulnerable to having
> conflicts between transactions opened against the two different dataset
> objects.
>
> ---
> A. Soroka
> The University of Virginia Library
>
>
> > On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com>
> wrote:
> >
> > I have a repeating memory leak in TDB in my web application (
> >
> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
> > ).
> > It is caching RDF documents from internet, typically dbpedia ressources.
> >
> > It is not the use case described in "Fuseki/TDB memory leak for
> concurrent
> > updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as the
> > journal is empty after crash .
> >
> > A single Dataset object is used for the duration of the application, and
> I
> > suspect this is the root cause.
> > In the documentation,
> >
> >
> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
> >
> > it is not clear which use pattern is preferred and the reason why.
> >
> > You someone confirm that keeping a single Dataset object for the duration
> > of the application is bad ?
>
>
--
Jean-Marc Vanel
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
http://deductions-software.com/
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Memory leak in TDB using a single Dataset object
Posted by "A. Soroka" <aj...@virginia.edu>.
> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com> wrote:
>
> In the documentation,
>
> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>
> it is not clear which use pattern is preferred and the reason why.
The first pattern shows a single dataset object being shared between threads, each of which operates a transaction against that object, and the second pattern is introduced with "or create a dataset object on the thread (the case above is preferred):”.
As to why, I am not familiar enough with TDB to be sure, but there is a comment on the second pattern "Each thread has a separate dataset object; these safely share the same storage but have independent transactions.” that would seem to indicate that the second pattern is vulnerable to having conflicts between transactions opened against the two different dataset objects.
---
A. Soroka
The University of Virginia Library
> On Feb 3, 2016, at 5:13 AM, Jean-Marc Vanel <je...@gmail.com> wrote:
>
> I have a repeating memory leak in TDB in my web application (
> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms_play/README.md
> ).
> It is caching RDF documents from internet, typically dbpedia ressources.
>
> It is not the use case described in "Fuseki/TDB memory leak for concurrent
> updates/queries" https://issues.apache.org/jira/browse/JENA-689 , as the
> journal is empty after crash .
>
> A single Dataset object is used for the duration of the application, and I
> suspect this is the root cause.
> In the documentation,
>
> https://jena.apache.org/documentation/tdb/tdb_transactions.html#multi-threaded-use
>
> it is not clear which use pattern is preferred and the reason why.
>
> You someone confirm that keeping a single Dataset object for the duration
> of the application is bad ?