You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Bill Roberts <bi...@swirrl.com> on 2012/04/01 23:41:01 UTC
Problem with Fuseki HTTP graph protocol and transactions
Hi all
I've been trying to find a simple repeatable example that might reproduce the database corruption problems I mentioned last week. Not sure this is exactly the same thing, but I suspect it is related.
The essence of it is that running a SELECT query while a PUT to /data is in progress seems to cause problems.
Unless you think I have missed something obvious, I'll add a ticket to the issue tracker.
Steps to reproduce (details of my set up are at the bottom of the email)
1. create empty dir for TDB and start fuseki with that directory as the tdb:location
2. PUT a small file to the graph protocol endpoint
curl -v -H "Content-Type: text/turtle" --upload-file small.ttl http://localhost:3030/crashtest/data?graph=http://test1
3. Run a count of all triples
select (count(*) as ?c) where {?s ?p ?o}
Answer in my case is 25 (as expected)
4. PUT a big file
curl -v -H "Content-Type: text/turtle" --upload-file big.ttl http://localhost:3030/crashtest/data?graph=http://test2
(big enough that it takes at least several seconds to load, so you have time to run some other stuff. My example was about 200,000 triples)
5. Before it finishes, run the count query another 2 or 3 times.
It comes back with 25 each time. So far so good.
6. After the big file load is finished (check for 201 Created in log), run the count again.
This is where the problem is evident: the count still shows 25, when it should show 200,000 or so.
(Probably not significant, but my small test file has a few blank nodes in it. The big file does not).
ls -l of the TDB dir shows lots of data still in nodes.dat-jrnl. The log includes the line:
"WARN TDB :: Transaction not active: 5"
(full copy of the log below)
Going through the same procedure without running the COUNTs mentioned in stage 5, then everything goes smoothly.
I'd be interested to hear if anyone else can reproduce this - and of course to hear what you think might be wrong!
Many thanks
Bill
Details:
OS: Macosx 10.6.8
fuseki-server --version
------------------------------
Jena: VERSION: 2.7.0-incubating
Jena: BUILD_DATE: 2011-12-14T14:54:09+0000
ARQ: VERSION: 2.9.0-incubating
ARQ: BUILD_DATE: 2011-12-14T15:04:27+0000
TDB: VERSION: 0.9.0-incubating
TDB: BUILD_DATE: 2012-02-29T19:39:52+0000
Fuseki: VERSION: 0.2.2-incubating-SNAPSHOT
Fuseki: BUILD_DATE: 20120330-0505
java -version
------------------
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
Config file:
---------------
@prefix tdb: <http://jena.hpl.hp.com/2008/tdb#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#> .
@prefix fuseki: <http://jena.apache.org/fuseki#> .
[] rdf:type fuseki:Server ;
# Services available. Only explicitly listed services are configured.
# If there is a service description not linked from this list, it is ignored.
fuseki:services (
<#service1>
) .
[] ja:loadClass "com.hp.hpl.jena.tdb.TDB" .
tdb:DatasetTDB rdfs:subClassOf ja:RDFDataset .
tdb:GraphTDB rdfs:subClassOf ja:Model .
<#service1> rdf:type fuseki:Service ;
fuseki:name "crashtest" ; # http://host:port/blah
fuseki:serviceQuery "query" ; # SPARQL query service
fuseki:serviceUpdate "update" ; # SPARQL update service
fuseki:serviceReadWriteGraphStore "data" ; # SPARQL Graph store protocol (read and write)
fuseki:dataset <#dataset-blah> ;
.
<#dataset-blah> rdf:type tdb:DatasetTDB ;
tdb:location "/Users/bill/tdb/crashtest" ;
# Query timeout on this dataset (1s, 1000 milliseconds)
ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "10000" ] ;
tdb:unionDefaultGraph true ;
fuseki log:
--------------
20:23:51 INFO Config :: Configuration file: test.ttl
20:23:51 INFO Config :: Service: <file:///Users/bill/code/fuseki-0.2.2/test.ttl#service1>
20:23:51 INFO Config :: name = crashtest
20:23:51 INFO Config :: query = /crashtest/query
20:23:51 INFO Config :: update = /crashtest/update
20:23:51 INFO Config :: graphStore(RW) = /crashtest/data
20:23:52 INFO Server :: Dataset path = /crashtest
20:23:52 INFO Server :: Fuseki 0.2.2-incubating-SNAPSHOT 20120330-0505
20:23:52 INFO Server :: Jetty 7.x.y-SNAPSHOT
20:23:52 INFO Server :: Started 2012/04/01 20:23:52 BST on port 3030
20:24:15 INFO Fuseki :: [1] PUT http://localhost:3030/crashtest/data?graph=http://test1
20:24:16 INFO Fuseki :: [1] 201 Created
20:24:43 INFO Fuseki :: [2] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
20:24:43 INFO Fuseki :: [2] Query = select (count(*) as ?c) where {?s ?p ?o}
20:24:43 INFO Fuseki :: [2] OK/select
20:24:43 INFO Fuseki :: [2] 200 OK
20:29:12 INFO Fuseki :: [3] PUT http://localhost:3030/crashtest/data?graph=http://test2
20:29:14 INFO Fuseki :: [4] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
20:29:14 INFO Fuseki :: [4] Query = select (count(*) as ?c) where {?s ?p ?o}
20:29:14 INFO Fuseki :: [4] OK/select
20:29:14 INFO Fuseki :: [4] 200 OK
20:29:18 INFO Fuseki :: [5] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
20:29:18 INFO Fuseki :: [5] Query = select (count(*) as ?c) where {?s ?p ?o}
20:29:18 INFO Fuseki :: [5] OK/select
20:29:18 INFO Fuseki :: [5] 200 OK
20:29:28 WARN TDB :: Transaction not active: 5
20:29:28 INFO Fuseki :: [3] 201 Created
20:29:28 INFO Fuseki :: [6] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
20:29:28 INFO Fuseki :: [6] Query = select (count(*) as ?c) where {?s ?p ?o}
20:29:28 INFO Fuseki :: [6] OK/select
20:29:28 INFO Fuseki :: [6] 200 OK
ls -l crashtest
------------------
drwxr-xr-x 31 bill bill 1054 1 Apr 20:24 ./
drwxr-xr-x 15 bill bill 510 1 Apr 20:23 ../
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GOSP.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GOSP.idn
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GPOS.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GPOS.idn
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GSPO.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GSPO.idn
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSP.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSP.idn
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 OSPG.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSPG.idn
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POS.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POS.idn
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 POSG.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POSG.idn
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPO.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPO.idn
-rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 SPOG.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPOG.idn
-rw-r--r-- 1 bill bill 0 1 Apr 20:24 journal.jrnl
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:24 node2id.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:24 node2id.idn
-rw-r--r-- 1 bill bill 2485 1 Apr 20:24 nodes.dat
-rw-r--r-- 1 bill bill 5596812 1 Apr 20:29 nodes.dat-jrnl
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefix2id.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefix2id.idn
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefixIdx.dat
-rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefixIdx.idn
-rw-r--r-- 1 bill bill 0 1 Apr 20:23 prefixes.dat
-rw-r--r-- 1 bill bill 0 1 Apr 20:24 prefixes.dat-jrnl
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Bill Roberts <bi...@swirrl.com>.
Andy
Thanks very much for the quick work. I will give the new version a try tomorrow and let you know how I get on.
Cheers
Bill
On 3 Apr 2012, at 17:19, Andy Seaborne wrote:
> Bill, Paul,
> cc Bernie
>
> Fix applied which should be picked up by tonight's build cycle.
>
> Bill - thanks for the details, I managed to recreate the situation
>
> Paul - I haven't managed to exactly recreate the situation from your description but this would explain it. The fact I can't recreate is just because I haven't hit the right timing sequence (I'm fairly sure that, while the concurrency hole looks big when looking at the code, JITting and the possibility of putting values on the stack means the timing hole is very much reduced).
>
> Bernie - FYI - this does not change the fact the best use of Datasets is one per thread.
>
> Andy
>
>
> On 02/04/12 21:20, Bill Roberts wrote:
>> Great, sounds promising. Now in JIRA: https://issues.apache.org/jira/browse/JENA-230
>>
>> thanks
>>
>> On 2 Apr 2012, at 20:19, Andy Seaborne wrote:
>>
>>> On 02/04/12 20:00, Bill Roberts wrote:
>>>> Ah, sorry, misread your previous mail re stage 3 queries.
>>>>
>>>> No, the number of queries at this point doesn't seem to make any difference, as long as the PUT has finished.
>>>>
>>>
>>> Great - and I have a theory I need to test out.
>>>
>>> (If it's a valid theory, this is the same root cause as Paul Gearon's problems despite them looking different to me initially).
>>>
>>> Andy
>>>
>>>>
>>>> On 2 Apr 2012, at 19:37, Andy Seaborne wrote:
>>>>
>>>>> On 02/04/12 19:33, Bill Roberts wrote:
>>>>>> Hi Andy
>>>>>> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>>>>>>
>>>>>>> Hi Bill,
>>>>>>>
>>>>>>> Sorry for the delay in replying. It was too sunny.
>>>>>>>
>>>>>>
>>>>>> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>>>>>>
>>>>>>> Please add it to JIRA.
>>>>>>
>>>>>> Will do.
>>>>>>
>>>>>>>
>>>>>>> There's nothing I can see that
>>>>>>>
>>>>>>> (how much RAM?)
>>>>>>
>>>>>> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>>>>>>
>>>>>>>
>>>>>>> big.ttl is the imd-2010-imd-score.ttl file?
>>>>>>>
>>>>>> I've tried it with a few different files (including that one) - same effect with each.
>>>>>>
>>>>>>>
>>>>>>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>>>>>>
>>>>>> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
>>>>>
>>>>> stage 3 is the queries after little the PUT - I'm guessing these make no difference if little PUT has finished.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks very much
>>>>>>
>>>>>> Bill
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Andy Seaborne <an...@apache.org>.
Bill, Paul,
cc Bernie
Fix applied which should be picked up by tonight's build cycle.
Bill - thanks for the details, I managed to recreate the situation
Paul - I haven't managed to exactly recreate the situation from your
description but this would explain it. The fact I can't recreate is
just because I haven't hit the right timing sequence (I'm fairly sure
that, while the concurrency hole looks big when looking at the code,
JITting and the possibility of putting values on the stack means the
timing hole is very much reduced).
Bernie - FYI - this does not change the fact the best use of Datasets is
one per thread.
Andy
On 02/04/12 21:20, Bill Roberts wrote:
> Great, sounds promising. Now in JIRA: https://issues.apache.org/jira/browse/JENA-230
>
> thanks
>
> On 2 Apr 2012, at 20:19, Andy Seaborne wrote:
>
>> On 02/04/12 20:00, Bill Roberts wrote:
>>> Ah, sorry, misread your previous mail re stage 3 queries.
>>>
>>> No, the number of queries at this point doesn't seem to make any difference, as long as the PUT has finished.
>>>
>>
>> Great - and I have a theory I need to test out.
>>
>> (If it's a valid theory, this is the same root cause as Paul Gearon's problems despite them looking different to me initially).
>>
>> Andy
>>
>>>
>>> On 2 Apr 2012, at 19:37, Andy Seaborne wrote:
>>>
>>>> On 02/04/12 19:33, Bill Roberts wrote:
>>>>> Hi Andy
>>>>> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>>>>>
>>>>>> Hi Bill,
>>>>>>
>>>>>> Sorry for the delay in replying. It was too sunny.
>>>>>>
>>>>>
>>>>> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>>>>>
>>>>>> Please add it to JIRA.
>>>>>
>>>>> Will do.
>>>>>
>>>>>>
>>>>>> There's nothing I can see that
>>>>>>
>>>>>> (how much RAM?)
>>>>>
>>>>> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>>>>>
>>>>>>
>>>>>> big.ttl is the imd-2010-imd-score.ttl file?
>>>>>>
>>>>> I've tried it with a few different files (including that one) - same effect with each.
>>>>>
>>>>>>
>>>>>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>>>>>
>>>>> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
>>>>
>>>> stage 3 is the queries after little the PUT - I'm guessing these make no difference if little PUT has finished.
>>>>
>>>> Andy
>>>>
>>>>
>>>>>
>>>>> Thanks very much
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>
>>
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Bill Roberts <bi...@swirrl.com>.
Great, sounds promising. Now in JIRA: https://issues.apache.org/jira/browse/JENA-230
thanks
On 2 Apr 2012, at 20:19, Andy Seaborne wrote:
> On 02/04/12 20:00, Bill Roberts wrote:
>> Ah, sorry, misread your previous mail re stage 3 queries.
>>
>> No, the number of queries at this point doesn't seem to make any difference, as long as the PUT has finished.
>>
>
> Great - and I have a theory I need to test out.
>
> (If it's a valid theory, this is the same root cause as Paul Gearon's problems despite them looking different to me initially).
>
> Andy
>
>>
>> On 2 Apr 2012, at 19:37, Andy Seaborne wrote:
>>
>>> On 02/04/12 19:33, Bill Roberts wrote:
>>>> Hi Andy
>>>> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>>>>
>>>>> Hi Bill,
>>>>>
>>>>> Sorry for the delay in replying. It was too sunny.
>>>>>
>>>>
>>>> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>>>>
>>>>> Please add it to JIRA.
>>>>
>>>> Will do.
>>>>
>>>>>
>>>>> There's nothing I can see that
>>>>>
>>>>> (how much RAM?)
>>>>
>>>> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>>>>
>>>>>
>>>>> big.ttl is the imd-2010-imd-score.ttl file?
>>>>>
>>>> I've tried it with a few different files (including that one) - same effect with each.
>>>>
>>>>>
>>>>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>>>>
>>>> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
>>>
>>> stage 3 is the queries after little the PUT - I'm guessing these make no difference if little PUT has finished.
>>>
>>> Andy
>>>
>>>
>>>>
>>>> Thanks very much
>>>>
>>>> Bill
>>>>
>>>
>>
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Andy Seaborne <an...@apache.org>.
On 02/04/12 20:00, Bill Roberts wrote:
> Ah, sorry, misread your previous mail re stage 3 queries.
>
> No, the number of queries at this point doesn't seem to make any difference, as long as the PUT has finished.
>
Great - and I have a theory I need to test out.
(If it's a valid theory, this is the same root cause as Paul Gearon's
problems despite them looking different to me initially).
Andy
>
> On 2 Apr 2012, at 19:37, Andy Seaborne wrote:
>
>> On 02/04/12 19:33, Bill Roberts wrote:
>>> Hi Andy
>>> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>>>
>>>> Hi Bill,
>>>>
>>>> Sorry for the delay in replying. It was too sunny.
>>>>
>>>
>>> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>>>
>>>> Please add it to JIRA.
>>>
>>> Will do.
>>>
>>>>
>>>> There's nothing I can see that
>>>>
>>>> (how much RAM?)
>>>
>>> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>>>
>>>>
>>>> big.ttl is the imd-2010-imd-score.ttl file?
>>>>
>>> I've tried it with a few different files (including that one) - same effect with each.
>>>
>>>>
>>>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>>>
>>> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
>>
>> stage 3 is the queries after little the PUT - I'm guessing these make no difference if little PUT has finished.
>>
>> Andy
>>
>>
>>>
>>> Thanks very much
>>>
>>> Bill
>>>
>>
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Bill Roberts <bi...@swirrl.com>.
Ah, sorry, misread your previous mail re stage 3 queries.
No, the number of queries at this point doesn't seem to make any difference, as long as the PUT has finished.
On 2 Apr 2012, at 19:37, Andy Seaborne wrote:
> On 02/04/12 19:33, Bill Roberts wrote:
>> Hi Andy
>> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>>
>>> Hi Bill,
>>>
>>> Sorry for the delay in replying. It was too sunny.
>>>
>>
>> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>>
>>> Please add it to JIRA.
>>
>> Will do.
>>
>>>
>>> There's nothing I can see that
>>>
>>> (how much RAM?)
>>
>> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>>
>>>
>>> big.ttl is the imd-2010-imd-score.ttl file?
>>>
>> I've tried it with a few different files (including that one) - same effect with each.
>>
>>>
>>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>>
>> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
>
> stage 3 is the queries after little the PUT - I'm guessing these make no difference if little PUT has finished.
>
> Andy
>
>
>>
>> Thanks very much
>>
>> Bill
>>
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Andy Seaborne <an...@apache.org>.
On 02/04/12 19:33, Bill Roberts wrote:
> Hi Andy
> On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
>
>> Hi Bill,
>>
>> Sorry for the delay in replying. It was too sunny.
>>
>
> No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>>
>> Please add it to JIRA.
>
> Will do.
>
>>
>> There's nothing I can see that
>>
>> (how much RAM?)
>
> 16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>
>>
>> big.ttl is the imd-2010-imd-score.ttl file?
>>
> I've tried it with a few different files (including that one) - same effect with each.
>
>>
>> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
>
> Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
stage 3 is the queries after little the PUT - I'm guessing these make no
difference if little PUT has finished.
Andy
>
> Thanks very much
>
> Bill
>
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Bill Roberts <bi...@swirrl.com>.
Hi Andy
On 2 Apr 2012, at 14:23, Andy Seaborne wrote:
> Hi Bill,
>
> Sorry for the delay in replying. It was too sunny.
>
No problem! I am all for appreciating a bit of sunniness when the chance presents itself.
>
> Please add it to JIRA.
Will do.
>
> There's nothing I can see that
>
> (how much RAM?)
16 GB on the server for the problems that I reported last week. 2GB on my Mac where this specific test was run (and a fair bit of that used by other software).
>
> big.ttl is the imd-2010-imd-score.ttl file?
>
I've tried it with a few different files (including that one) - same effect with each.
>
> Does it have to be 2 queries at stage 3? Does one have the same effect? None?
Just tested a bit more: with a single query during the PUT, it worked correctly 4 out of 5 times and failed once. With two queries during the PUT, the problem occurred in all tests (tried about 10 times with 3 different 'big' files).
Thanks very much
Bill
Re: Problem with Fuseki HTTP graph protocol and transactions
Posted by Andy Seaborne <an...@apache.org>.
Hi Bill,
Sorry for the delay in replying. It was too sunny.
On 01/04/12 22:41, Bill Roberts wrote:
> Hi all
>
> I've been trying to find a simple repeatable example that might reproduce the database corruption problems I mentioned last week. Not sure this is exactly the same thing, but I suspect it is related.
>
> The essence of it is that running a SELECT query while a PUT to /data is in progress seems to cause problems.
>
> Unless you think I have missed something obvious, I'll add a ticket to the issue tracker.
Please add it to JIRA.
There's nothing I can see that
(how much RAM?)
big.ttl is the imd-2010-imd-score.ttl file?
>
> Steps to reproduce (details of my set up are at the bottom of the email)
>
> 1. create empty dir for TDB and start fuseki with that directory as the tdb:location
>
> 2. PUT a small file to the graph protocol endpoint
> curl -v -H "Content-Type: text/turtle" --upload-file small.ttl http://localhost:3030/crashtest/data?graph=http://test1
>
> 3. Run a count of all triples
> select (count(*) as ?c) where {?s ?p ?o}
> Answer in my case is 25 (as expected)
Does it have to be 2 queries at stage 3? Does one have the same effect?
None?
> 4. PUT a big file
> curl -v -H "Content-Type: text/turtle" --upload-file big.ttl http://localhost:3030/crashtest/data?graph=http://test2
> (big enough that it takes at least several seconds to load, so you have time to run some other stuff. My example was about 200,000 triples)
>
> 5. Before it finishes, run the count query another 2 or 3 times.
> It comes back with 25 each time. So far so good.
>
> 6. After the big file load is finished (check for 201 Created in log), run the count again.
> This is where the problem is evident: the count still shows 25, when it should show 200,000 or so.
>
> (Probably not significant, but my small test file has a few blank nodes in it. The big file does not).
>
> ls -l of the TDB dir shows lots of data still in nodes.dat-jrnl. The log includes the line:
> "WARN TDB :: Transaction not active: 5"
> (full copy of the log below)
>
> Going through the same procedure without running the COUNTs mentioned in stage 5, then everything goes smoothly.
>
> I'd be interested to hear if anyone else can reproduce this - and of course to hear what you think might be wrong!
>
> Many thanks
>
> Bill
Thanks
Andy
>
> Details:
>
>
>
> OS: Macosx 10.6.8
>
> fuseki-server --version
> ------------------------------
> Jena: VERSION: 2.7.0-incubating
> Jena: BUILD_DATE: 2011-12-14T14:54:09+0000
> ARQ: VERSION: 2.9.0-incubating
> ARQ: BUILD_DATE: 2011-12-14T15:04:27+0000
> TDB: VERSION: 0.9.0-incubating
> TDB: BUILD_DATE: 2012-02-29T19:39:52+0000
> Fuseki: VERSION: 0.2.2-incubating-SNAPSHOT
> Fuseki: BUILD_DATE: 20120330-0505
>
> java -version
> ------------------
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
>
>
> Config file:
> ---------------
> @prefix tdb:<http://jena.hpl.hp.com/2008/tdb#> .
> @prefix rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
> @prefix rdfs:<http://www.w3.org/2000/01/rdf-schema#> .
> @prefix ja:<http://jena.hpl.hp.com/2005/11/Assembler#> .
> @prefix fuseki:<http://jena.apache.org/fuseki#> .
>
> [] rdf:type fuseki:Server ;
> # Services available. Only explicitly listed services are configured.
> # If there is a service description not linked from this list, it is ignored.
> fuseki:services (
> <#service1>
> ) .
>
>
> [] ja:loadClass "com.hp.hpl.jena.tdb.TDB" .
> tdb:DatasetTDB rdfs:subClassOf ja:RDFDataset .
> tdb:GraphTDB rdfs:subClassOf ja:Model .
>
> <#service1> rdf:type fuseki:Service ;
> fuseki:name "crashtest" ; # http://host:port/blah
> fuseki:serviceQuery "query" ; # SPARQL query service
> fuseki:serviceUpdate "update" ; # SPARQL update service
> fuseki:serviceReadWriteGraphStore "data" ; # SPARQL Graph store protocol (read and write)
> fuseki:dataset<#dataset-blah> ;
> .
>
>
> <#dataset-blah> rdf:type tdb:DatasetTDB ;
> tdb:location "/Users/bill/tdb/crashtest" ;
> # Query timeout on this dataset (1s, 1000 milliseconds)
> ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "10000" ] ;
> tdb:unionDefaultGraph true ;
>
> ://
> fuseki log:
> --------------
>
> 20:23:51 INFO Config :: Configuration file: test.ttl
> 20:23:51 INFO Config :: Service:<file:///Users/bill/code/fuseki-0.2.2/test.ttl#service1>
> 20:23:51 INFO Config :: name = crashtest
> 20:23:51 INFO Config :: query = /crashtest/query
> 20:23:51 INFO Config :: update = /crashtest/update
> 20:23:51 INFO Config :: graphStore(RW) = /crashtest/data
> 20:23:52 INFO Server :: Dataset path = /crashtest
> 20:23:52 INFO Server :: Fuseki 0.2.2-incubating-SNAPSHOT 20120330-0505
> 20:23:52 INFO Server :: Jetty 7.x.y-SNAPSHOT
> 20:23:52 INFO Server :: Started 2012/04/01 20:23:52 BST on port 3030
> 20:24:15 INFO Fuseki :: [1] PUT http://localhost:3030/crashtest/data?graph=http://test1
> 20:24:16 INFO Fuseki :: [1] 201 Created
> 20:24:43 INFO Fuseki :: [2] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
> 20:24:43 INFO Fuseki :: [2] Query = select (count(*) as ?c) where {?s ?p ?o}
> 20:24:43 INFO Fuseki :: [2] OK/select
> 20:24:43 INFO Fuseki :: [2] 200 OK
> 20:29:12 INFO Fuseki :: [3] PUT http://localhost:3030/crashtest/data?graph=http://test2
> 20:29:14 INFO Fuseki :: [4] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
> 20:29:14 INFO Fuseki :: [4] Query = select (count(*) as ?c) where {?s ?p ?o}
> 20:29:14 INFO Fuseki :: [4] OK/select
> 20:29:14 INFO Fuseki :: [4] 200 OK
> 20:29:18 INFO Fuseki :: [5] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
> 20:29:18 INFO Fuseki :: [5] Query = select (count(*) as ?c) where {?s ?p ?o}
> 20:29:18 INFO Fuseki :: [5] OK/select
> 20:29:18 INFO Fuseki :: [5] 200 OK
> 20:29:28 WARN TDB :: Transaction not active: 5
> 20:29:28 INFO Fuseki :: [3] 201 Created
> 20:29:28 INFO Fuseki :: [6] GET http://localhost:3030/crashtest/query?query=select+%28count%28*%29+as+%3Fc%29+where+%7B%3Fs+%3Fp+%3Fo%7D+&output=text&stylesheet=%2Fxml-to-html.xsl
> 20:29:28 INFO Fuseki :: [6] Query = select (count(*) as ?c) where {?s ?p ?o}
> 20:29:28 INFO Fuseki :: [6] OK/select
> 20:29:28 INFO Fuseki :: [6] 200 OK
>
> ls -l crashtest
> ------------------
> drwxr-xr-x 31 bill bill 1054 1 Apr 20:24 ./
> drwxr-xr-x 15 bill bill 510 1 Apr 20:23 ../
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GOSP.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GOSP.idn
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GPOS.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GPOS.idn
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 GSPO.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 GSPO.idn
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSP.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSP.idn
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 OSPG.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 OSPG.idn
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POS.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POS.idn
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 POSG.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 POSG.idn
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPO.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPO.idn
> -rw-r--r-- 1 bill bill 16777216 1 Apr 20:29 SPOG.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 SPOG.idn
> -rw-r--r-- 1 bill bill 0 1 Apr 20:24 journal.jrnl
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:24 node2id.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:24 node2id.idn
> -rw-r--r-- 1 bill bill 2485 1 Apr 20:24 nodes.dat
> -rw-r--r-- 1 bill bill 5596812 1 Apr 20:29 nodes.dat-jrnl
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefix2id.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefix2id.idn
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefixIdx.dat
> -rw-r--r-- 1 bill bill 8388608 1 Apr 20:23 prefixIdx.idn
> -rw-r--r-- 1 bill bill 0 1 Apr 20:23 prefixes.dat
> -rw-r--r-- 1 bill bill 0 1 Apr 20:24 prefixes.dat-jrnl
>