You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Rob Vesse <rv...@dotnetrdf.org> on 2014/06/03 10:43:04 UTC
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Mark
Answers inline:
On 02/06/2014 19:14, "Mark Feblowitz" <Ma...@comcast.net> wrote:
>Ok -
>
>It’s another day… after a weekend’s rest. And now I have a few
>observations and many questions:
>
>So, a bit more about what I’m doing (much of it likely naive):
>
>I create a local ontmodel, create several statements in that model,
>convert the model to a string, wrap it with INSERT DATA { }, then post it
>to Fuseki using an Update Request:
>
> UpdateRequest ur = new UpdateRequest();
> ur.add(payloadContent);
> UpdateProcessor up = UpdateExecutionFactory.createRemote(ur, service);
> up.execute();
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> }
If your graphs are relatively small then it might be more efficient to
look at using DatasetAccessor
(https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
tasetAccessor.html) to PUT the entire updated graph to the server via the
Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
<http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
DatasetAccessor accessor =
DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data");
accessor.putModel(model, "http://example.org/graph");
>
>
>I have standalone code that sends SPARQL queries to the same Fuseki
>server, using the usual methods.
>
>So, before I go deeper into the questions, it occurs to me that I don’t
>necessarily need to post remotely, and removing that overhead would be an
>obvious improvement . I’m wondering whether my update code could interact
>directly with TDB while also having the content be queriable via Fuseki.
>I recall trying that a while ago and noting that direct posts to TDB were
>not available for query until restarting Fuseki. Was that a (repaired)
>bug or a feature?
DO NOT do this!
TDB is only designed for single JVM usage and attaching multiple JVMs to a
dataset runs the risk of irrevocably corrupting your dataset. Work is
actually under way to add functionality to TDB to prevent users doing this
because we've seen far too many cases of data corruption resulting from
this.
>
>If I can’t do this, do you know of any projects that have created a
>server that can be run with my application and share the TDB connection?
You want a triple store that can run in both embedded and remote mode
simultaneously? No I don't know of any
There are stores that support non-HTTP protocols (ODBC, custom protocol
buffer based etc) that may offer better performance and allow you to have
multiple applications talking to the server over different protocols but
they still require a single server. Most of these are however commercial
options but we can provide your our recommendations if you want them.
>
>Assuming that I do stick with remote updates, I have some questions:
>
>First, note that I create a new Update processor for each posted update.
>I don’t see any up.close() so I’m guessing I’m not creating any leakage
>here. Would it be better to create one update processor and reuse it
>across updates?
Yes each remote update request is a self contained HTTP request with no
response body expected so the connection is closed automatically after the
request is made and the response headers read.
>
>Next, I’m seeing a lot of notices like the following in the Fuseki
>console:
>
>13:40:07 INFO [19125] 204 No Content (21 ms)
>13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19126] 204 No Content (13 ms)
>13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19127] 204 No Content (16 ms)
>13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19128] 204 No Content (14 ms)
>13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>13:40:07 INFO [19129] 204 No Content (8 ms)
>13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>
>I can't tell whether this reflects
> 1) a high overall number of update calls from my code, or
> 2) that the Fuseki side processes each of the triples for an update as
>an http post.
It reflects both
A remote update request is executed by making a HTTP POST to the /update
endpoint seen in your logs
204 No Content is the standard Fuseki response to a successful update
since there is not expected to be any response body from an update
>
>If it’s the latter, I don’t think I can batch up more updates to improve
>overall performance.
>
>I guess I can tell by inserting delays between my posts and capturing
>timestamps in my log that can be compared against what I’m seeing in the
>Fuseki console. I’ll try that next.
The number in square brackets is the request ID and is included in the
response as a Fuseki-Request-ID header. Returned headers aren't exposed
to the ARQ code per se but you can up the debugging level for the
org.apache.http packages for your client application to see detailed HTTP
traces in the logs of the communications between your application and
Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
more recent versions of ARQ include more recent versions of Apache
HttpClient which need you to turn the log level all the way up to TRACE
while older ARQ versions only needed the log level up to DEBUG to see the
traces.
Rob
>
>I guess that’s it for the questions for now.
>
>Any recommendations?
>
>Thanks again,
>
>Mark
>
>
>
>On May 31, 2014, at 12:22 PM, Andy Seaborne <an...@apache.org> wrote:
>
>> Hi Mark,
>>
>> The long running query is quite significant.
>>
>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>> That’s a good idea.
>>>
>>> One improvement I’ve already made was to relocate the DB to local
>>> disk - having it on a shared filesystem is an even worse idea.
>>>
>>> The updates tend to be on the order of 5-20 triples at a time.
>>
>> If you could batch up changes, that will help for all sorts of reasons.
>> c.f. autocommit and JDBC where many small changes run really slowly.
>>
>> This is part of the issue - write transactions have a significant fixed
>> cost that you get for even a (theoretical) transaction of no changes. It
>> has to write a few bytes and do a disk-sync. Reads continue during this
>> time but longer write times means there is less chance of system being
>> able to write the journal to the main database. JENA-567 may help but
>> isn't faster (it's slower) but it saves memory.
>>
>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>read-centric.
>>
>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>of transaction state. Multiple times for multiple indexes. So 5
>>triples of change get every little shared block effect and the memory
>>footprint is disproportionally large.
>>
>> <thinking out loud id=1>
>>
>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>blocks at 8k (they have in different block managers = files) would be
>>worth experimenting with.
>>
>> </thinking out loud>
>>
>> <thinking out loud id=2>
>>
>> We could provide some netty/mima/... based server that did the moral
>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>it is the nature of HTTP; it is cautious wording because I haven't
>>observed it myself - Jetty locally, together with careful streaming of
>>results seems to be quite effective. Fast encoding of results would be
>>good for both.
>>
>> </thinking out loud>
>>
>>> I believe I identified the worst culprit, and that was using
>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>> TransitiveReasoner. My guess is that the longish query chain over a
>>> large triplestore, using the Owl reasoner was leading to very long
>>> query times and lots of memory consumption. Do you think that’s a
>>> reasonable guess?
>>
>> That does look right. Long running queries, or the effect of an
>>intense stream of small back-to-back queries combined with the update
>>pattern, leave no time for the system to flush the journal back to the
>>main database. This leads to memory usage and eventually OOME.
>>
>>> How I reached that conclusion was to kill the non-responsive (even
>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>> DB, with many triples). After that, both the small query and the
>>> multi-join query responded quite quickly.
>>>
>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>> control of the submissions.
>>
>> That should at least prove whether this discussion has correctly
>>diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>("disgraceful") behaviour as the load reaches system saturation. It
>>ought to be more graceful but, fundamentally, its always going to be
>>possible to flood a system, any system, with more work than it is
>>capable of.
>>
>> Andy
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>>> Mark,
>>>>
>>>> How big are the updates?
>>>>
>>>> An SSD for the database and the journal will help.
>>>>
>>>> Every transaction is a commit, and a commit is a disk operation to
>>>> ensure the commit record is permanent. That is not cheap with a
>>>> rotational disk (seek time), and much better with an SSD.
>>>>
>>>> If you are driving Fuseki as hard as possible, something will break
>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>> well as being more defensive.
>>>>
>>>> Andy
>>>>
>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>> Mark
>>>>>
>>>>> This sounds like the same problem described in
>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>
>>>>> TL;DR
>>>>>
>>>>> For a system with no quiescent periods continually receiving
>>>>> updates the in-memory journal continues to expand until such time
>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>> journal is a write ahead log and is first written to disk (you
>>>>> will lose at most the data from the transaction that encountered
>>>>> the OOM). Therefore once the system is restarted the journal
>>>>> will be replayed and flushed.
>>>>>
>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>> experimental feature that may mitigate this and see
>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>> tracking the work to remove this limitation
>>>>>
>>>>> Rob
>>
>
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Posted by Andy Seaborne <an...@apache.org>.
On 25/07/14 17:44, Mark Feblowitz wrote:
> I use the latter - separate submissions of a single update request containing several statements :
>
>>
>> INSERT DATA { ...
>> ...
>> ...
>> } ;
>
>
> So it seems that I could consider buffering up a number of these within the same update request.
>
> Out of curiosity, does an update request get submitted to the server as one “payload” and processed at the server, or is the set broken down into individual INSERT DATA requests that are sent across the wire? Or as individual INSERT DATA requests, one per statement (one per “…”)?
ARQ does not add any buffering internally - one DatasetAccessor or one
UpdateProcessor.execute is one HTTP request.
>
> What would be the scope of the transaction? Per update request or per INSERT DATA?
HTTP Request (= update request).
>
> How are transaction failures reported?
HTTP response codes
(note that there aren't any system generated aborts so failures are due
to bad input (400) or system bugs (500))
Andy
>
> Thanks,
>
> Mark
>
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Posted by Mark Feblowitz <Ma...@comcast.net>.
I use the latter - separate submissions of a single update request containing several statements :
>
> INSERT DATA { ...
> ...
> ...
> } ;
So it seems that I could consider buffering up a number of these within the same update request.
Out of curiosity, does an update request get submitted to the server as one “payload” and processed at the server, or is the set broken down into individual INSERT DATA requests that are sent across the wire? Or as individual INSERT DATA requests, one per statement (one per “…”)?
What would be the scope of the transaction? Per update request or per INSERT DATA?
How are transaction failures reported?
Thanks,
Mark
On Jul 25, 2014, at 6:31 AM, Andy Seaborne <an...@apache.org> wrote:
> On 24/07/14 19:08, Mark Feblowitz wrote:
>> Ok -
>>
>> I tried the DatasetAccessor approach and discovered the subtlety in the phrase "PUT the entire updated graph to the server.” What I found is that, upon each update, the graph subset that I’m sending as an update becomes the complete graph.
>>
>
> yes - that's what HTTP PUT does.
>
> POST is append (i.e. add triples)
>
>> My use case is somewhat unusual: I have a set of statements that I’m wrapping with
>>
>> "INSERT DATA { " + statements + "}";
>>
>> and using the UpdateProcessor from UpdateExecutionFactory to send to Fuseki.
>>
>> From the logs I’m assuming that the UpdateProcessor iterates through the statements and posts them individually, leading to a potentially large number of individual, rapid-fire updates (?).
>>
>> So what I attempted to do is to read those statements into a model and post them using the DataAccessor. Bad idea.
>>
>> What I’d like to have done is to send across a set of statements and have the set be processed as an update. And, thinking ahead, I’m fairly certain that the update should be treated as a transaction (across the set of statements).
>>
>> Is anything like that supported?
>
>
> If you send them one at a time, they execute one at a time.
>
> You can join them together by a single SPARQL Update request ...
>
> INSERT DATA { ... } ;
> INSERT DATA { ... } ;
> INSERT DATA { ... } ;
>
> or probably just
>
> INSERT DATA { ...
> ...
> ...
> } ;
>
> or build up a UpdateRequest with a series of Update
>
> UpdateRequest == whole HTTP SPARQL Update
> Update = individual operations within the request - in syntax, separated by ";"
>
>>
>> BTW - SSD for TDB does seem to have addressed my current update rate problem - I’ve not seen Fuseki lockups since moving to SSD. But my current code is a prototype; I expect there’ll be an increase in update rates that will eventually nullify this approach.
>>
>> Thanks,
>>
>> Mark
>>
>>
>>
>> On Jun 3, 2014, at 5:54 AM, Andy Seaborne <an...@apache.org> wrote:
>>
>>> On 03/06/14 09:43, Rob Vesse wrote:
>>>> Mark
>>>>
>>>> Answers inline:
>>>>
>>>> On 02/06/2014 19:14, "Mark Feblowitz" <Ma...@comcast.net> wrote:
>>>>
>>>>> Ok -
>>>>>
>>>>> It’s another day… after a weekend’s rest. And now I have a few
>>>>> observations and many questions:
>>>>>
>>>>> So, a bit more about what I’m doing (much of it likely naive):
>>>>>
>>>>> I create a local ontmodel, create several statements in that model,
>>>>> convert the model to a string, wrap it with INSERT DATA { }, then post it
>>>>> to Fuseki using an Update Request:
>>>>>
>>>>> UpdateRequest ur = new UpdateRequest();
>>>>> ur.add(payloadContent);
>>>>> UpdateProcessor up = UpdateExecutionFactory.createRemote(ur, service);
>>>>> up.execute();
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> } finally {
>>>>> }
>>>>
>>>> If your graphs are relatively small then it might be more efficient to
>>>> look at using DatasetAccessor
>>>> (https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
>>>> tasetAccessor.html) to PUT the entire updated graph to the server via the
>>>> Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
>>>> <http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
>>>>
>>>> DatasetAccessor accessor =
>>>> DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data");
>>>> accessor.putModel(model, "http://example.org/graph");
>>>>
>>>>>
>>>>>
>>>>> I have standalone code that sends SPARQL queries to the same Fuseki
>>>>> server, using the usual methods.
>>>>>
>>>>> So, before I go deeper into the questions, it occurs to me that I don’t
>>>>> necessarily need to post remotely, and removing that overhead would be an
>>>>> obvious improvement . I’m wondering whether my update code could interact
>>>>> directly with TDB while also having the content be queriable via Fuseki.
>>>>> I recall trying that a while ago and noting that direct posts to TDB were
>>>>> not available for query until restarting Fuseki. Was that a (repaired)
>>>>> bug or a feature?
>>>>
>>>> DO NOT do this!
>>>
>>> It is not a bug (would you edit the files for MySQL bypassing the MySQL engine?)
>>>
>>>> TDB is only designed for single JVM usage and attaching multiple JVMs to a
>>>> dataset runs the risk of irrevocably corrupting your dataset. Work is
>>>> actually under way to add functionality to TDB to prevent users doing this
>>>> because we've seen far too many cases of data corruption resulting from
>>>> this.
>>>>
>>>>
>>>>>
>>>>> If I can’t do this, do you know of any projects that have created a
>>>>> server that can be run with my application and share the TDB connection?
>>>>
>>>> You want a triple store that can run in both embedded and remote mode
>>>> simultaneously? No I don't know of any
>>>>
>>>> There are stores that support non-HTTP protocols (ODBC, custom protocol
>>>> buffer based etc) that may offer better performance and allow you to have
>>>> multiple applications talking to the server over different protocols but
>>>> they still require a single server. Most of these are however commercial
>>>> options but we can provide your our recommendations if you want them.
>>>>
>>>>>
>>>>> Assuming that I do stick with remote updates, I have some questions:
>>>>>
>>>>> First, note that I create a new Update processor for each posted update.
>>>>> I don’t see any up.close() so I’m guessing I’m not creating any leakage
>>>>> here. Would it be better to create one update processor and reuse it
>>>>> across updates?
>>>>
>>>> Yes each remote update request is a self contained HTTP request with no
>>>> response body expected so the connection is closed automatically after the
>>>> request is made and the response headers read.
>>>>
>>>>>
>>>>> Next, I’m seeing a lot of notices like the following in the Fuseki
>>>>> console:
>>>>>
>>>>> 13:40:07 INFO [19125] 204 No Content (21 ms)
>>>>> 13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>>>>> 13:40:07 INFO [19126] 204 No Content (13 ms)
>>>>> 13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>>>>> 13:40:07 INFO [19127] 204 No Content (16 ms)
>>>>> 13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>>>>> 13:40:07 INFO [19128] 204 No Content (14 ms)
>>>>> 13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>>>>> 13:40:07 INFO [19129] 204 No Content (8 ms)
>>>>> 13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>>>>>
>>>>> I can't tell whether this reflects
>>>>> 1) a high overall number of update calls from my code, or
>>>>> 2) that the Fuseki side processes each of the triples for an update as
>>>>> an http post.
>>>
>>> The client API sends one SPARQL update for each SPARQL update the app requests. It does not break up a INSERT DATA nor does it combine them.
>>>
>>>>
>>>> It reflects both
>>>>
>>>> A remote update request is executed by making a HTTP POST to the /update
>>>> endpoint seen in your logs
>>>>
>>>> 204 No Content is the standard Fuseki response to a successful update
>>>> since there is not expected to be any response body from an update
>>>>
>>>>>
>>>>> If it’s the latter, I don’t think I can batch up more updates to improve
>>>>> overall performance.
>>>>>
>>>>> I guess I can tell by inserting delays between my posts and capturing
>>>>> timestamps in my log that can be compared against what I’m seeing in the
>>>>> Fuseki console. I’ll try that next.
>>>>
>>>> The number in square brackets is the request ID and is included in the
>>>> response as a Fuseki-Request-ID header. Returned headers aren't exposed
>>>> to the ARQ code per se but you can up the debugging level for the
>>>> org.apache.http packages for your client application to see detailed HTTP
>>>> traces in the logs of the communications between your application and
>>>> Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
>>>> more recent versions of ARQ include more recent versions of Apache
>>>> HttpClient which need you to turn the log level all the way up to TRACE
>>>> while older ARQ versions only needed the log level up to DEBUG to see the
>>>> traces.
>>>
>>> I think if you run the server "--verbose" it'll print out yet more details as well.
>>>
>>> Andy
>>>
>>>>
>>>> Rob
>>>>
>>>>>
>>>>> I guess that’s it for the questions for now.
>>>>>
>>>>> Any recommendations?
>>>>>
>>>>> Thanks again,
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>> On May 31, 2014, at 12:22 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>>
>>>>>> Hi Mark,
>>>>>>
>>>>>> The long running query is quite significant.
>>>>>>
>>>>>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>>>>>> That’s a good idea.
>>>>>>>
>>>>>>> One improvement I’ve already made was to relocate the DB to local
>>>>>>> disk - having it on a shared filesystem is an even worse idea.
>>>>>>>
>>>>>>> The updates tend to be on the order of 5-20 triples at a time.
>>>>>>
>>>>>> If you could batch up changes, that will help for all sorts of reasons.
>>>>>> c.f. autocommit and JDBC where many small changes run really slowly.
>>>>>>
>>>>>> This is part of the issue - write transactions have a significant fixed
>>>>>> cost that you get for even a (theoretical) transaction of no changes. It
>>>>>> has to write a few bytes and do a disk-sync. Reads continue during this
>>>>>> time but longer write times means there is less chance of system being
>>>>>> able to write the journal to the main database. JENA-567 may help but
>>>>>> isn't faster (it's slower) but it saves memory.
>>>>>>
>>>>>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>>>>> read-centric.
>>>>>>
>>>>>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>>>>> of transaction state. Multiple times for multiple indexes. So 5
>>>>>> triples of change get every little shared block effect and the memory
>>>>>> footprint is disproportionally large.
>>>>>>
>>>>>> <thinking out loud id=1>
>>>>>>
>>>>>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>>>>> blocks at 8k (they have in different block managers = files) would be
>>>>>> worth experimenting with.
>>>>>>
>>>>>> </thinking out loud>
>>>>>>
>>>>>> <thinking out loud id=2>
>>>>>>
>>>>>> We could provide some netty/mima/... bassqrted server that did the moral
>>>>>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>>>>> said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>>>>> it is the nature of HTTP; it is cautious wording because I haven't
>>>>>> observed it myself - Jetty locally, together with careful streaming of
>>>>>> results seems to be quite effective. Fast encoding of results would be
>>>>>> good for both.
>>>>>>
>>>>>> </thinking out loud>
>>>>>>
>>>>>>> I believe I identified the worst culprit, and that was using
>>>>>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>>>>>> TransitiveReasoner. My guess is that the longish query chain over a
>>>>>>> large triplestore, using the Owl reasoner was leading to very long
>>>>>>> query times and lots of memory consumption. Do you think that’s a
>>>>>>> reasonable guess?
>>>>>>
>>>>>> That does look right. Long running queries, or the effect of an
>>>>>> intense stream of small back-to-back queries combined with the update
>>>>>> pattern, leave no time for the system to flush the journal back to the
>>>>>> main database. This leads to memory usage and eventually OOME.
>>>>>>
>>>>>>> How I reached that conclusion was to kill the non-responsive (even
>>>>>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>>>>>> DB, with many triples). After that, both the small query and the
>>>>>>> multi-join query responded quite quickly.
>>>>>>>
>>>>>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>>>>>> control of the submissions.
>>>>>>
>>>>>> That should at least prove whether this discussion has correctly
>>>>>> diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>>>>> ("disgraceful") behaviour as the load reaches system saturation. It
>>>>>> ought to be more graceful but, fundamentally, its always going to be
>>>>>> possible to flood a system, any system, with more work than it is
>>>>>> capable of.
>>>>>>
>>>>>> Andy
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>>>>
>>>>>>>> Mark,
>>>>>>>>
>>>>>>>> How big are the updates?
>>>>>>>>
>>>>>>>> An SSD for the database and the journal will help.
>>>>>>>>
>>>>>>>> Every transaction is a commit, and a commit is a disk operation to
>>>>>>>> ensure the commit record is permanent. That is not cheap with a
>>>>>>>> rotational disk (seek time), and much better with an SSD.
>>>>>>>>
>>>>>>>> If you are driving Fuseki as hard as possible, something will break
>>>>>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>>>>>> well as being more defensive.
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>> This sounds like the same problem described in
>>>>>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>>>>>
>>>>>>>>> TL;DR
>>>>>>>>>
>>>>>>>>> For a system with no quiescent periods continually receiving
>>>>>>>>> updates the in-memory journal continues to expand until such time
>>>>>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>>>>>> journal is a write ahead log and is first written to disk (you
>>>>>>>>> will lose at most the data from the transaction that encountered
>>>>>>>>> the OOM). Therefore once the system is restarted the journal
>>>>>>>>> will be replayed and flushed.
>>>>>>>>>
>>>>>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>>>>>> experimental feature that may mitigate this and see
>>>>>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>>>>>> tracking the work to remove this limitation
>>>>>>>>>
>>>>>>>>> Rob
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Posted by Andy Seaborne <an...@apache.org>.
On 24/07/14 19:08, Mark Feblowitz wrote:
> Ok -
>
> I tried the DatasetAccessor approach and discovered the subtlety in the phrase "PUT the entire updated graph to the server.” What I found is that, upon each update, the graph subset that I’m sending as an update becomes the complete graph.
>
yes - that's what HTTP PUT does.
POST is append (i.e. add triples)
> My use case is somewhat unusual: I have a set of statements that I’m wrapping with
>
> "INSERT DATA { " + statements + "}";
>
> and using the UpdateProcessor from UpdateExecutionFactory to send to Fuseki.
>
> From the logs I’m assuming that the UpdateProcessor iterates through the statements and posts them individually, leading to a potentially large number of individual, rapid-fire updates (?).
>
> So what I attempted to do is to read those statements into a model and post them using the DataAccessor. Bad idea.
>
> What I’d like to have done is to send across a set of statements and have the set be processed as an update. And, thinking ahead, I’m fairly certain that the update should be treated as a transaction (across the set of statements).
>
> Is anything like that supported?
If you send them one at a time, they execute one at a time.
You can join them together by a single SPARQL Update request ...
INSERT DATA { ... } ;
INSERT DATA { ... } ;
INSERT DATA { ... } ;
or probably just
INSERT DATA { ...
...
...
} ;
or build up a UpdateRequest with a series of Update
UpdateRequest == whole HTTP SPARQL Update
Update = individual operations within the request - in syntax,
separated by ";"
>
> BTW - SSD for TDB does seem to have addressed my current update rate problem - I’ve not seen Fuseki lockups since moving to SSD. But my current code is a prototype; I expect there’ll be an increase in update rates that will eventually nullify this approach.
>
> Thanks,
>
> Mark
>
>
>
> On Jun 3, 2014, at 5:54 AM, Andy Seaborne <an...@apache.org> wrote:
>
>> On 03/06/14 09:43, Rob Vesse wrote:
>>> Mark
>>>
>>> Answers inline:
>>>
>>> On 02/06/2014 19:14, "Mark Feblowitz" <Ma...@comcast.net> wrote:
>>>
>>>> Ok -
>>>>
>>>> It’s another day… after a weekend’s rest. And now I have a few
>>>> observations and many questions:
>>>>
>>>> So, a bit more about what I’m doing (much of it likely naive):
>>>>
>>>> I create a local ontmodel, create several statements in that model,
>>>> convert the model to a string, wrap it with INSERT DATA { }, then post it
>>>> to Fuseki using an Update Request:
>>>>
>>>> UpdateRequest ur = new UpdateRequest();
>>>> ur.add(payloadContent);
>>>> UpdateProcessor up = UpdateExecutionFactory.createRemote(ur, service);
>>>> up.execute();
>>>> } catch (Exception e) {
>>>> e.printStackTrace();
>>>> } finally {
>>>> }
>>>
>>> If your graphs are relatively small then it might be more efficient to
>>> look at using DatasetAccessor
>>> (https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
>>> tasetAccessor.html) to PUT the entire updated graph to the server via the
>>> Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
>>> <http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
>>>
>>> DatasetAccessor accessor =
>>> DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data");
>>> accessor.putModel(model, "http://example.org/graph");
>>>
>>>>
>>>>
>>>> I have standalone code that sends SPARQL queries to the same Fuseki
>>>> server, using the usual methods.
>>>>
>>>> So, before I go deeper into the questions, it occurs to me that I don’t
>>>> necessarily need to post remotely, and removing that overhead would be an
>>>> obvious improvement . I’m wondering whether my update code could interact
>>>> directly with TDB while also having the content be queriable via Fuseki.
>>>> I recall trying that a while ago and noting that direct posts to TDB were
>>>> not available for query until restarting Fuseki. Was that a (repaired)
>>>> bug or a feature?
>>>
>>> DO NOT do this!
>>
>> It is not a bug (would you edit the files for MySQL bypassing the MySQL engine?)
>>
>>> TDB is only designed for single JVM usage and attaching multiple JVMs to a
>>> dataset runs the risk of irrevocably corrupting your dataset. Work is
>>> actually under way to add functionality to TDB to prevent users doing this
>>> because we've seen far too many cases of data corruption resulting from
>>> this.
>>>
>>>
>>>>
>>>> If I can’t do this, do you know of any projects that have created a
>>>> server that can be run with my application and share the TDB connection?
>>>
>>> You want a triple store that can run in both embedded and remote mode
>>> simultaneously? No I don't know of any
>>>
>>> There are stores that support non-HTTP protocols (ODBC, custom protocol
>>> buffer based etc) that may offer better performance and allow you to have
>>> multiple applications talking to the server over different protocols but
>>> they still require a single server. Most of these are however commercial
>>> options but we can provide your our recommendations if you want them.
>>>
>>>>
>>>> Assuming that I do stick with remote updates, I have some questions:
>>>>
>>>> First, note that I create a new Update processor for each posted update.
>>>> I don’t see any up.close() so I’m guessing I’m not creating any leakage
>>>> here. Would it be better to create one update processor and reuse it
>>>> across updates?
>>>
>>> Yes each remote update request is a self contained HTTP request with no
>>> response body expected so the connection is closed automatically after the
>>> request is made and the response headers read.
>>>
>>>>
>>>> Next, I’m seeing a lot of notices like the following in the Fuseki
>>>> console:
>>>>
>>>> 13:40:07 INFO [19125] 204 No Content (21 ms)
>>>> 13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>>>> 13:40:07 INFO [19126] 204 No Content (13 ms)
>>>> 13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>>>> 13:40:07 INFO [19127] 204 No Content (16 ms)
>>>> 13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>>>> 13:40:07 INFO [19128] 204 No Content (14 ms)
>>>> 13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>>>> 13:40:07 INFO [19129] 204 No Content (8 ms)
>>>> 13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>>>>
>>>> I can't tell whether this reflects
>>>> 1) a high overall number of update calls from my code, or
>>>> 2) that the Fuseki side processes each of the triples for an update as
>>>> an http post.
>>
>> The client API sends one SPARQL update for each SPARQL update the app requests. It does not break up a INSERT DATA nor does it combine them.
>>
>>>
>>> It reflects both
>>>
>>> A remote update request is executed by making a HTTP POST to the /update
>>> endpoint seen in your logs
>>>
>>> 204 No Content is the standard Fuseki response to a successful update
>>> since there is not expected to be any response body from an update
>>>
>>>>
>>>> If it’s the latter, I don’t think I can batch up more updates to improve
>>>> overall performance.
>>>>
>>>> I guess I can tell by inserting delays between my posts and capturing
>>>> timestamps in my log that can be compared against what I’m seeing in the
>>>> Fuseki console. I’ll try that next.
>>>
>>> The number in square brackets is the request ID and is included in the
>>> response as a Fuseki-Request-ID header. Returned headers aren't exposed
>>> to the ARQ code per se but you can up the debugging level for the
>>> org.apache.http packages for your client application to see detailed HTTP
>>> traces in the logs of the communications between your application and
>>> Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
>>> more recent versions of ARQ include more recent versions of Apache
>>> HttpClient which need you to turn the log level all the way up to TRACE
>>> while older ARQ versions only needed the log level up to DEBUG to see the
>>> traces.
>>
>> I think if you run the server "--verbose" it'll print out yet more details as well.
>>
>> Andy
>>
>>>
>>> Rob
>>>
>>>>
>>>> I guess that’s it for the questions for now.
>>>>
>>>> Any recommendations?
>>>>
>>>> Thanks again,
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>> On May 31, 2014, at 12:22 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>
>>>>> Hi Mark,
>>>>>
>>>>> The long running query is quite significant.
>>>>>
>>>>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>>>>> That’s a good idea.
>>>>>>
>>>>>> One improvement I’ve already made was to relocate the DB to local
>>>>>> disk - having it on a shared filesystem is an even worse idea.
>>>>>>
>>>>>> The updates tend to be on the order of 5-20 triples at a time.
>>>>>
>>>>> If you could batch up changes, that will help for all sorts of reasons.
>>>>> c.f. autocommit and JDBC where many small changes run really slowly.
>>>>>
>>>>> This is part of the issue - write transactions have a significant fixed
>>>>> cost that you get for even a (theoretical) transaction of no changes. It
>>>>> has to write a few bytes and do a disk-sync. Reads continue during this
>>>>> time but longer write times means there is less chance of system being
>>>>> able to write the journal to the main database. JENA-567 may help but
>>>>> isn't faster (it's slower) but it saves memory.
>>>>>
>>>>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>>>> read-centric.
>>>>>
>>>>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>>>> of transaction state. Multiple times for multiple indexes. So 5
>>>>> triples of change get every little shared block effect and the memory
>>>>> footprint is disproportionally large.
>>>>>
>>>>> <thinking out loud id=1>
>>>>>
>>>>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>>>> blocks at 8k (they have in different block managers = files) would be
>>>>> worth experimenting with.
>>>>>
>>>>> </thinking out loud>
>>>>>
>>>>> <thinking out loud id=2>
>>>>>
>>>>> We could provide some netty/mima/... bassqrted server that did the moral
>>>>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>>>> said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>>>> it is the nature of HTTP; it is cautious wording because I haven't
>>>>> observed it myself - Jetty locally, together with careful streaming of
>>>>> results seems to be quite effective. Fast encoding of results would be
>>>>> good for both.
>>>>>
>>>>> </thinking out loud>
>>>>>
>>>>>> I believe I identified the worst culprit, and that was using
>>>>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>>>>> TransitiveReasoner. My guess is that the longish query chain over a
>>>>>> large triplestore, using the Owl reasoner was leading to very long
>>>>>> query times and lots of memory consumption. Do you think that’s a
>>>>>> reasonable guess?
>>>>>
>>>>> That does look right. Long running queries, or the effect of an
>>>>> intense stream of small back-to-back queries combined with the update
>>>>> pattern, leave no time for the system to flush the journal back to the
>>>>> main database. This leads to memory usage and eventually OOME.
>>>>>
>>>>>> How I reached that conclusion was to kill the non-responsive (even
>>>>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>>>>> DB, with many triples). After that, both the small query and the
>>>>>> multi-join query responded quite quickly.
>>>>>>
>>>>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>>>>> control of the submissions.
>>>>>
>>>>> That should at least prove whether this discussion has correctly
>>>>> diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>>>> ("disgraceful") behaviour as the load reaches system saturation. It
>>>>> ought to be more graceful but, fundamentally, its always going to be
>>>>> possible to flood a system, any system, with more work than it is
>>>>> capable of.
>>>>>
>>>>> Andy
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>>>
>>>>>>> Mark,
>>>>>>>
>>>>>>> How big are the updates?
>>>>>>>
>>>>>>> An SSD for the database and the journal will help.
>>>>>>>
>>>>>>> Every transaction is a commit, and a commit is a disk operation to
>>>>>>> ensure the commit record is permanent. That is not cheap with a
>>>>>>> rotational disk (seek time), and much better with an SSD.
>>>>>>>
>>>>>>> If you are driving Fuseki as hard as possible, something will break
>>>>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>>>>> well as being more defensive.
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> This sounds like the same problem described in
>>>>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>>>>
>>>>>>>> TL;DR
>>>>>>>>
>>>>>>>> For a system with no quiescent periods continually receiving
>>>>>>>> updates the in-memory journal continues to expand until such time
>>>>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>>>>> journal is a write ahead log and is first written to disk (you
>>>>>>>> will lose at most the data from the transaction that encountered
>>>>>>>> the OOM). Therefore once the system is restarted the journal
>>>>>>>> will be replayed and flushed.
>>>>>>>>
>>>>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>>>>> experimental feature that may mitigate this and see
>>>>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>>>>> tracking the work to remove this limitation
>>>>>>>>
>>>>>>>> Rob
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
>
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Posted by Mark Feblowitz <Ma...@comcast.net>.
Ok -
I tried the DatasetAccessor approach and discovered the subtlety in the phrase "PUT the entire updated graph to the server.” What I found is that, upon each update, the graph subset that I’m sending as an update becomes the complete graph.
My use case is somewhat unusual: I have a set of statements that I’m wrapping with
"INSERT DATA { " + statements + "}";
and using the UpdateProcessor from UpdateExecutionFactory to send to Fuseki.
From the logs I’m assuming that the UpdateProcessor iterates through the statements and posts them individually, leading to a potentially large number of individual, rapid-fire updates (?).
So what I attempted to do is to read those statements into a model and post them using the DataAccessor. Bad idea.
What I’d like to have done is to send across a set of statements and have the set be processed as an update. And, thinking ahead, I’m fairly certain that the update should be treated as a transaction (across the set of statements).
Is anything like that supported?
BTW - SSD for TDB does seem to have addressed my current update rate problem - I’ve not seen Fuseki lockups since moving to SSD. But my current code is a prototype; I expect there’ll be an increase in update rates that will eventually nullify this approach.
Thanks,
Mark
On Jun 3, 2014, at 5:54 AM, Andy Seaborne <an...@apache.org> wrote:
> On 03/06/14 09:43, Rob Vesse wrote:
>> Mark
>>
>> Answers inline:
>>
>> On 02/06/2014 19:14, "Mark Feblowitz" <Ma...@comcast.net> wrote:
>>
>>> Ok -
>>>
>>> It’s another day… after a weekend’s rest. And now I have a few
>>> observations and many questions:
>>>
>>> So, a bit more about what I’m doing (much of it likely naive):
>>>
>>> I create a local ontmodel, create several statements in that model,
>>> convert the model to a string, wrap it with INSERT DATA { }, then post it
>>> to Fuseki using an Update Request:
>>>
>>> UpdateRequest ur = new UpdateRequest();
>>> ur.add(payloadContent);
>>> UpdateProcessor up = UpdateExecutionFactory.createRemote(ur, service);
>>> up.execute();
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> } finally {
>>> }
>>
>> If your graphs are relatively small then it might be more efficient to
>> look at using DatasetAccessor
>> (https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
>> tasetAccessor.html) to PUT the entire updated graph to the server via the
>> Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
>> <http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
>>
>> DatasetAccessor accessor =
>> DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data");
>> accessor.putModel(model, "http://example.org/graph");
>>
>>>
>>>
>>> I have standalone code that sends SPARQL queries to the same Fuseki
>>> server, using the usual methods.
>>>
>>> So, before I go deeper into the questions, it occurs to me that I don’t
>>> necessarily need to post remotely, and removing that overhead would be an
>>> obvious improvement . I’m wondering whether my update code could interact
>>> directly with TDB while also having the content be queriable via Fuseki.
>>> I recall trying that a while ago and noting that direct posts to TDB were
>>> not available for query until restarting Fuseki. Was that a (repaired)
>>> bug or a feature?
>>
>> DO NOT do this!
>
> It is not a bug (would you edit the files for MySQL bypassing the MySQL engine?)
>
>> TDB is only designed for single JVM usage and attaching multiple JVMs to a
>> dataset runs the risk of irrevocably corrupting your dataset. Work is
>> actually under way to add functionality to TDB to prevent users doing this
>> because we've seen far too many cases of data corruption resulting from
>> this.
>>
>>
>>>
>>> If I can’t do this, do you know of any projects that have created a
>>> server that can be run with my application and share the TDB connection?
>>
>> You want a triple store that can run in both embedded and remote mode
>> simultaneously? No I don't know of any
>>
>> There are stores that support non-HTTP protocols (ODBC, custom protocol
>> buffer based etc) that may offer better performance and allow you to have
>> multiple applications talking to the server over different protocols but
>> they still require a single server. Most of these are however commercial
>> options but we can provide your our recommendations if you want them.
>>
>>>
>>> Assuming that I do stick with remote updates, I have some questions:
>>>
>>> First, note that I create a new Update processor for each posted update.
>>> I don’t see any up.close() so I’m guessing I’m not creating any leakage
>>> here. Would it be better to create one update processor and reuse it
>>> across updates?
>>
>> Yes each remote update request is a self contained HTTP request with no
>> response body expected so the connection is closed automatically after the
>> request is made and the response headers read.
>>
>>>
>>> Next, I’m seeing a lot of notices like the following in the Fuseki
>>> console:
>>>
>>> 13:40:07 INFO [19125] 204 No Content (21 ms)
>>> 13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>>> 13:40:07 INFO [19126] 204 No Content (13 ms)
>>> 13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>>> 13:40:07 INFO [19127] 204 No Content (16 ms)
>>> 13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>>> 13:40:07 INFO [19128] 204 No Content (14 ms)
>>> 13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>>> 13:40:07 INFO [19129] 204 No Content (8 ms)
>>> 13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>>>
>>> I can't tell whether this reflects
>>> 1) a high overall number of update calls from my code, or
>>> 2) that the Fuseki side processes each of the triples for an update as
>>> an http post.
>
> The client API sends one SPARQL update for each SPARQL update the app requests. It does not break up a INSERT DATA nor does it combine them.
>
>>
>> It reflects both
>>
>> A remote update request is executed by making a HTTP POST to the /update
>> endpoint seen in your logs
>>
>> 204 No Content is the standard Fuseki response to a successful update
>> since there is not expected to be any response body from an update
>>
>>>
>>> If it’s the latter, I don’t think I can batch up more updates to improve
>>> overall performance.
>>>
>>> I guess I can tell by inserting delays between my posts and capturing
>>> timestamps in my log that can be compared against what I’m seeing in the
>>> Fuseki console. I’ll try that next.
>>
>> The number in square brackets is the request ID and is included in the
>> response as a Fuseki-Request-ID header. Returned headers aren't exposed
>> to the ARQ code per se but you can up the debugging level for the
>> org.apache.http packages for your client application to see detailed HTTP
>> traces in the logs of the communications between your application and
>> Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
>> more recent versions of ARQ include more recent versions of Apache
>> HttpClient which need you to turn the log level all the way up to TRACE
>> while older ARQ versions only needed the log level up to DEBUG to see the
>> traces.
>
> I think if you run the server "--verbose" it'll print out yet more details as well.
>
> Andy
>
>>
>> Rob
>>
>>>
>>> I guess that’s it for the questions for now.
>>>
>>> Any recommendations?
>>>
>>> Thanks again,
>>>
>>> Mark
>>>
>>>
>>>
>>> On May 31, 2014, at 12:22 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>>> Hi Mark,
>>>>
>>>> The long running query is quite significant.
>>>>
>>>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>>>> That’s a good idea.
>>>>>
>>>>> One improvement I’ve already made was to relocate the DB to local
>>>>> disk - having it on a shared filesystem is an even worse idea.
>>>>>
>>>>> The updates tend to be on the order of 5-20 triples at a time.
>>>>
>>>> If you could batch up changes, that will help for all sorts of reasons.
>>>> c.f. autocommit and JDBC where many small changes run really slowly.
>>>>
>>>> This is part of the issue - write transactions have a significant fixed
>>>> cost that you get for even a (theoretical) transaction of no changes. It
>>>> has to write a few bytes and do a disk-sync. Reads continue during this
>>>> time but longer write times means there is less chance of system being
>>>> able to write the journal to the main database. JENA-567 may help but
>>>> isn't faster (it's slower) but it saves memory.
>>>>
>>>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>>> read-centric.
>>>>
>>>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>>> of transaction state. Multiple times for multiple indexes. So 5
>>>> triples of change get every little shared block effect and the memory
>>>> footprint is disproportionally large.
>>>>
>>>> <thinking out loud id=1>
>>>>
>>>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>>> blocks at 8k (they have in different block managers = files) would be
>>>> worth experimenting with.
>>>>
>>>> </thinking out loud>
>>>>
>>>> <thinking out loud id=2>
>>>>
>>>> We could provide some netty/mima/... bassqrted server that did the moral
>>>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>>> said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>>> it is the nature of HTTP; it is cautious wording because I haven't
>>>> observed it myself - Jetty locally, together with careful streaming of
>>>> results seems to be quite effective. Fast encoding of results would be
>>>> good for both.
>>>>
>>>> </thinking out loud>
>>>>
>>>>> I believe I identified the worst culprit, and that was using
>>>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>>>> TransitiveReasoner. My guess is that the longish query chain over a
>>>>> large triplestore, using the Owl reasoner was leading to very long
>>>>> query times and lots of memory consumption. Do you think that’s a
>>>>> reasonable guess?
>>>>
>>>> That does look right. Long running queries, or the effect of an
>>>> intense stream of small back-to-back queries combined with the update
>>>> pattern, leave no time for the system to flush the journal back to the
>>>> main database. This leads to memory usage and eventually OOME.
>>>>
>>>>> How I reached that conclusion was to kill the non-responsive (even
>>>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>>>> DB, with many triples). After that, both the small query and the
>>>>> multi-join query responded quite quickly.
>>>>>
>>>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>>>> control of the submissions.
>>>>
>>>> That should at least prove whether this discussion has correctly
>>>> diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>>> ("disgraceful") behaviour as the load reaches system saturation. It
>>>> ought to be more graceful but, fundamentally, its always going to be
>>>> possible to flood a system, any system, with more work than it is
>>>> capable of.
>>>>
>>>> Andy
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mark
>>>>>
>>>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> How big are the updates?
>>>>>>
>>>>>> An SSD for the database and the journal will help.
>>>>>>
>>>>>> Every transaction is a commit, and a commit is a disk operation to
>>>>>> ensure the commit record is permanent. That is not cheap with a
>>>>>> rotational disk (seek time), and much better with an SSD.
>>>>>>
>>>>>> If you are driving Fuseki as hard as possible, something will break
>>>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>>>> well as being more defensive.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>>>> Mark
>>>>>>>
>>>>>>> This sounds like the same problem described in
>>>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>>>
>>>>>>> TL;DR
>>>>>>>
>>>>>>> For a system with no quiescent periods continually receiving
>>>>>>> updates the in-memory journal continues to expand until such time
>>>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>>>> journal is a write ahead log and is first written to disk (you
>>>>>>> will lose at most the data from the transaction that encountered
>>>>>>> the OOM). Therefore once the system is restarted the journal
>>>>>>> will be replayed and flushed.
>>>>>>>
>>>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>>>> experimental feature that may mitigate this and see
>>>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>>>> tracking the work to remove this limitation
>>>>>>>
>>>>>>> Rob
>>>>
>>>
>>
>>
>>
>>
>
Re: Jena/Fuseki/TDB Java heap space and OutOfMemory errors
Posted by Andy Seaborne <an...@apache.org>.
On 03/06/14 09:43, Rob Vesse wrote:
> Mark
>
> Answers inline:
>
> On 02/06/2014 19:14, "Mark Feblowitz" <Ma...@comcast.net> wrote:
>
>> Ok -
>>
>> It’s another day… after a weekend’s rest. And now I have a few
>> observations and many questions:
>>
>> So, a bit more about what I’m doing (much of it likely naive):
>>
>> I create a local ontmodel, create several statements in that model,
>> convert the model to a string, wrap it with INSERT DATA { }, then post it
>> to Fuseki using an Update Request:
>>
>> UpdateRequest ur = new UpdateRequest();
>> ur.add(payloadContent);
>> UpdateProcessor up = UpdateExecutionFactory.createRemote(ur, service);
>> up.execute();
>> } catch (Exception e) {
>> e.printStackTrace();
>> } finally {
>> }
>
> If your graphs are relatively small then it might be more efficient to
> look at using DatasetAccessor
> (https://jena.apache.org/documentation/javadoc/arq/com/hp/hpl/jena/query/Da
> tasetAccessor.html) to PUT the entire updated graph to the server via the
> Graph Store Protocol (SPARQL 1.1 Graph Store HTTP Protocol
> <http://www.w3.org/TR/sparql11-http-rdf-update/>) e.g.
>
> DatasetAccessor accessor =
> DatasetAccessorFactory.createHTTP("http://localhost:3030/ds/data");
> accessor.putModel(model, "http://example.org/graph");
>
>>
>>
>> I have standalone code that sends SPARQL queries to the same Fuseki
>> server, using the usual methods.
>>
>> So, before I go deeper into the questions, it occurs to me that I don’t
>> necessarily need to post remotely, and removing that overhead would be an
>> obvious improvement . I’m wondering whether my update code could interact
>> directly with TDB while also having the content be queriable via Fuseki.
>> I recall trying that a while ago and noting that direct posts to TDB were
>> not available for query until restarting Fuseki. Was that a (repaired)
>> bug or a feature?
>
> DO NOT do this!
It is not a bug (would you edit the files for MySQL bypassing the MySQL
engine?)
> TDB is only designed for single JVM usage and attaching multiple JVMs to a
> dataset runs the risk of irrevocably corrupting your dataset. Work is
> actually under way to add functionality to TDB to prevent users doing this
> because we've seen far too many cases of data corruption resulting from
> this.
>
>
>>
>> If I can’t do this, do you know of any projects that have created a
>> server that can be run with my application and share the TDB connection?
>
> You want a triple store that can run in both embedded and remote mode
> simultaneously? No I don't know of any
>
> There are stores that support non-HTTP protocols (ODBC, custom protocol
> buffer based etc) that may offer better performance and allow you to have
> multiple applications talking to the server over different protocols but
> they still require a single server. Most of these are however commercial
> options but we can provide your our recommendations if you want them.
>
>>
>> Assuming that I do stick with remote updates, I have some questions:
>>
>> First, note that I create a new Update processor for each posted update.
>> I don’t see any up.close() so I’m guessing I’m not creating any leakage
>> here. Would it be better to create one update processor and reuse it
>> across updates?
>
> Yes each remote update request is a self contained HTTP request with no
> response body expected so the connection is closed automatically after the
> request is made and the response headers read.
>
>>
>> Next, I’m seeing a lot of notices like the following in the Fuseki
>> console:
>>
>> 13:40:07 INFO [19125] 204 No Content (21 ms)
>> 13:40:07 INFO [19126] POST http://localhost:3030/km4sp/update
>> 13:40:07 INFO [19126] 204 No Content (13 ms)
>> 13:40:07 INFO [19127] POST http://localhost:3030/km4sp/update
>> 13:40:07 INFO [19127] 204 No Content (16 ms)
>> 13:40:07 INFO [19128] POST http://localhost:3030/km4sp/update
>> 13:40:07 INFO [19128] 204 No Content (14 ms)
>> 13:40:07 INFO [19129] POST http://localhost:3030/km4sp/update
>> 13:40:07 INFO [19129] 204 No Content (8 ms)
>> 13:40:07 INFO [19130] POST http://localhost:3030/km4sp/update
>>
>> I can't tell whether this reflects
>> 1) a high overall number of update calls from my code, or
>> 2) that the Fuseki side processes each of the triples for an update as
>> an http post.
The client API sends one SPARQL update for each SPARQL update the app
requests. It does not break up a INSERT DATA nor does it combine them.
>
> It reflects both
>
> A remote update request is executed by making a HTTP POST to the /update
> endpoint seen in your logs
>
> 204 No Content is the standard Fuseki response to a successful update
> since there is not expected to be any response body from an update
>
>>
>> If it’s the latter, I don’t think I can batch up more updates to improve
>> overall performance.
>>
>> I guess I can tell by inserting delays between my posts and capturing
>> timestamps in my log that can be compared against what I’m seeing in the
>> Fuseki console. I’ll try that next.
>
> The number in square brackets is the request ID and is included in the
> response as a Fuseki-Request-ID header. Returned headers aren't exposed
> to the ARQ code per se but you can up the debugging level for the
> org.apache.http packages for your client application to see detailed HTTP
> traces in the logs of the communications between your application and
> Fuseki. You'll need DEBUG or TRACE level depending on the version of ARQ,
> more recent versions of ARQ include more recent versions of Apache
> HttpClient which need you to turn the log level all the way up to TRACE
> while older ARQ versions only needed the log level up to DEBUG to see the
> traces.
I think if you run the server "--verbose" it'll print out yet more
details as well.
Andy
>
> Rob
>
>>
>> I guess that’s it for the questions for now.
>>
>> Any recommendations?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>> On May 31, 2014, at 12:22 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>>> Hi Mark,
>>>
>>> The long running query is quite significant.
>>>
>>> On 30/05/14 18:26, Mark Feblowitz wrote:
>>>> That’s a good idea.
>>>>
>>>> One improvement I’ve already made was to relocate the DB to local
>>>> disk - having it on a shared filesystem is an even worse idea.
>>>>
>>>> The updates tend to be on the order of 5-20 triples at a time.
>>>
>>> If you could batch up changes, that will help for all sorts of reasons.
>>> c.f. autocommit and JDBC where many small changes run really slowly.
>>>
>>> This is part of the issue - write transactions have a significant fixed
>>> cost that you get for even a (theoretical) transaction of no changes. It
>>> has to write a few bytes and do a disk-sync. Reads continue during this
>>> time but longer write times means there is less chance of system being
>>> able to write the journal to the main database. JENA-567 may help but
>>> isn't faster (it's slower) but it saves memory.
>>>
>>> Read transactions have near zero cost in TDB - Fuseki/TDB is
>>> read-centric.
>>>
>>> What's more, TDB block size is 8Kbytes so one change in a block is 8K
>>> of transaction state. Multiple times for multiple indexes. So 5
>>> triples of change get every little shared block effect and the memory
>>> footprint is disproportionally large.
>>>
>>> <thinking out loud id=1>
>>>
>>> A block size of 1 or 2k for the leaf blocks in TDB, leaving the branch
>>> blocks at 8k (they have in different block managers = files) would be
>>> worth experimenting with.
>>>
>>> </thinking out loud>
>>>
>>> <thinking out loud id=2>
>>>
>>> We could provide some netty/mima/... bassqrted server that did the moral
>>> equivalent of the SPARQL Protocol (cf jena-jdbc, jena-client). HTTP is
>>> said to be an appreciable cost. This is no judgement of Jetty/tomcat,
>>> it is the nature of HTTP; it is cautious wording because I haven't
>>> observed it myself - Jetty locally, together with careful streaming of
>>> results seems to be quite effective. Fast encoding of results would be
>>> good for both.
>>>
>>> </thinking out loud>
>>>
>>>> I believe I identified the worst culprit, and that was using
>>>> OWLFBRuleReasoner rather than RDFSExptRuleReasoner or
>>>> TransitiveReasoner. My guess is that the longish query chain over a
>>>> large triplestore, using the Owl reasoner was leading to very long
>>>> query times and lots of memory consumption. Do you think that’s a
>>>> reasonable guess?
>>>
>>> That does look right. Long running queries, or the effect of an
>>> intense stream of small back-to-back queries combined with the update
>>> pattern, leave no time for the system to flush the journal back to the
>>> main database. This leads to memory usage and eventually OOME.
>>>
>>>> How I reached that conclusion was to kill the non-responsive (even
>>>> for a small query) Fuseki and restart with RDFSExptRuleReasoner (same
>>>> DB, with many triples). After that, both the small query and the
>>>> multi-join query responded quite quickly.
>>>>
>>>> If necessary, I’ll try to throttle the posts, since I’m in complete
>>>> control of the submissions.
>>>
>>> That should at least prove whether this discussion has correctly
>>> diagnosed the interactions leading to OOME. What we have is ungrace-ful
>>> ("disgraceful") behaviour as the load reaches system saturation. It
>>> ought to be more graceful but, fundamentally, its always going to be
>>> possible to flood a system, any system, with more work than it is
>>> capable of.
>>>
>>> Andy
>>>>
>>>> Thanks,
>>>>
>>>> Mark
>>>>
>>>> On May 30, 2014, at 12:52 PM, Andy Seaborne <an...@apache.org> wrote:
>>>>
>>>>> Mark,
>>>>>
>>>>> How big are the updates?
>>>>>
>>>>> An SSD for the database and the journal will help.
>>>>>
>>>>> Every transaction is a commit, and a commit is a disk operation to
>>>>> ensure the commit record is permanent. That is not cheap with a
>>>>> rotational disk (seek time), and much better with an SSD.
>>>>>
>>>>> If you are driving Fuseki as hard as possible, something will break
>>>>> - the proposal in JENA-703 amounts to slowing the clients down as
>>>>> well as being more defensive.
>>>>>
>>>>> Andy
>>>>>
>>>>> On 30/05/14 15:39, Rob Vesse wrote:
>>>>>> Mark
>>>>>>
>>>>>> This sounds like the same problem described in
>>>>>> https://issues.apache.org/jira/browse/JENA-689
>>>>>>
>>>>>> TL;DR
>>>>>>
>>>>>> For a system with no quiescent periods continually receiving
>>>>>> updates the in-memory journal continues to expand until such time
>>>>>> as an OOM occurs. There will be little/no data loss because the
>>>>>> journal is a write ahead log and is first written to disk (you
>>>>>> will lose at most the data from the transaction that encountered
>>>>>> the OOM). Therefore once the system is restarted the journal
>>>>>> will be replayed and flushed.
>>>>>>
>>>>>> See https://issues.apache.org/jira/browse/JENA-567 for an
>>>>>> experimental feature that may mitigate this and see
>>>>>> https://issues.apache.org/jira/browse/JENA-703 for the issue
>>>>>> tracking the work to remove this limitation
>>>>>>
>>>>>> Rob
>>>
>>
>
>
>
>