You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by "Lebling, David (US SSA)" <da...@baesystems.com> on 2012/09/07 15:08:34 UTC

Model critical sections

I have a suite of webservices which use OWL ontologies stored as named graphs in an SDB database. I also have a set of Java objects that provide a POJO façade over ontology class instances. These instances import many or even all of the OWL ontologies, so I have an OntDocumentManager which points to cached versions of the ontologies.

When an ontology changes, I need to update it in the database, then re-cache it in the OntDocumentManager. During those operations (update ontology graph in database, update cached version in OntDocumentManager) I am within a try/finally block that (respectively) is a write lock or a read lock.

So a sketch (not the full implementation) of the code for writing and reading would be:

void update(String name, Model newModel) {
Model model = SDBFactory.connectNamedModel(store, name);
try {
                model.enterCriticalSection(false); // write
                model.removeAll();
                model.add(newModel);
}
finally {
                model.leaveCriticalSection();
}
}

void cache(OntDocumentManager mgr, String name) {
                Model model = SDBFactory.connectNamedModel(store, name);
                Model cachedModel = createModel(); // this method creates an OWL_MEM model with the doc mgr as its document manager, etc.
                try {
                                model.enterCriticalSection(true); // read
                                cachedModel.add(model);
                }
                finally {
                                model.leaveCriticalSection();
}
mgr.addModel(name, cachedModel, true);
}

What appears to happen is that these critical sections don't appear to inhibit other users (in my tests I have five) of the SDB database from modifying the stored graph. Readers of the graphs sometimes end up with empty graphs. I will sometimes end up with stored graphs that are empty, or inconsistent in some way that eventually corrupts the underlying tables in the database, requiring a complete reinitialization. I also get IllegalThreadState exceptions. Many different exceptions are thrown, with little obvious consistency.

Is critical section only good within a JVM? Do I need to use some sort of transaction instead?

Does anyone have any advice or ideas?

Dave

David Lebling
Principal Software Engineer
BAE Systems, ACDS Group


Re: using "or" in sparql

Posted by Francesco Panico <fp...@imolinfo.it>.
Hi

Your query selects all triples without  source, at first. If  the two
elements without  author have a source they will be discarded, because the
value added by "optional" clause have to respect "where clauses" anyway.

You should try to use UNION construct, because you want to retrieve a list
with
-item without a source
-item without author



2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>

> Hi,
>
> I'm having trouble with constructing a query that uses "or". I already
> tried optional, but it does not lead to the expected result. I'd be glad if
> someone could help me. I'd like to query for:
>
> all var bindings that have no author or no source.
>
> I tried:
>
> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>
> But it only computes the first part (source).
>
> If I have 3 items without a source and 2 others with no author, then I
> should get 5 results instead only the first 3. What am I doing wrong?
>
> Thanks in advance,
> Katja
>
>


-- 
ufficio:  0542 32640
mail:     fpanico@imolinfo.it
cell:      349 3508348
skype:  francies82@gmail.com

Re: using "or" in sparql

Posted by Katja Siegemund <Ka...@tu-dresden.de>.
Hi Francesco,

thanks a lot, it's working now, I also had to use SELECT DISTINCT to 
prevent duplicates.
The W3C Rec. for SPARQL is quite un-understandable in some cases ;)

Cheers,
Katja

Am 10.09.2012 12:29, schrieb Francesco Panico:
> Hi
>
> Your query selects all triples without  "source", at first. If  the two
> elements without  an "author" have a "source", they will be discarded,
> because the values added by "optional" clause have to respect "where
> clauses" anyway.
> You can check it.
>
> You can try to use UNION construct, because you want to retrieve a list
> with
> -item without a source
> -item without author
>
> SELECT ?r
> WHERE {?r a ro:FR.
>                   FILTER NOT EXISTS {?r ro:hasSource ?a}
>
> UNION{    ?r a ro:FR.
>                    FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>              }
> }
>
>
> I hope it is useful.
> Regards
> Francesco.
>
>
>
>
>
>
> 2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>
>
>> Hi,
>>
>> I'm having trouble with constructing a query that uses "or". I already
>> tried optional, but it does not lead to the expected result. I'd be glad if
>> someone could help me. I'd like to query for:
>>
>> all var bindings that have no author or no source.
>>
>> I tried:
>>
>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>
>> But it only computes the first part (source).
>>
>> If I have 3 items without a source and 2 others with no author, then I
>> should get 5 results instead only the first 3. What am I doing wrong?
>>
>> Thanks in advance,
>> Katja
>>
>>
>


-- 
Dipl.-Medieninf. Katja Siegemund
Research Assistant

Technische Universität Dresden
Department of Computer Science

Phone +49 (0)40 572 44 55 1
Email Katja.Siegemund@tu-dresden.de
WWW   http://st.inf.tu-dresden.de/


Re: using "or" in sparql

Posted by Francesco Panico <fp...@imolinfo.it>.
Yes, you can use multiple  "Union" in a sparql query.



2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>

> Hm, well, but how can I use UNION for multiple times? I have a lot of
> "or's" in my query :(
>
>
>
> Am 10.09.2012 12:29, schrieb Francesco Panico:
>
>> Hi
>>
>>
>> Your query selects all triples without  "source", at first. If  the two
>> elements without  an "author" have a "source", they will be discarded,
>> because the values added by "optional" clause have to respect "where
>> clauses" anyway.
>> You can check it.
>>
>> You can try to use UNION construct, because you want to retrieve a list
>> with
>> -item without a source
>> -item without author
>>
>> SELECT ?r
>> WHERE {?r a ro:FR.
>>                   FILTER NOT EXISTS {?r ro:hasSource ?a}
>>
>> UNION{    ?r a ro:FR.
>>                    FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>>              }
>> }
>>
>>
>> I hope it is useful.
>> Regards
>> Francesco.
>>
>>
>>
>>
>>
>>
>> 2012/9/10 Katja Siegemund <Katja.Siegemund@tu-dresden.de**>
>>
>>  Hi,
>>>
>>> I'm having trouble with constructing a query that uses "or". I already
>>> tried optional, but it does not lead to the expected result. I'd be glad
>>> if
>>> someone could help me. I'd like to query for:
>>>
>>> all var bindings that have no author or no source.
>>>
>>> I tried:
>>>
>>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>>
>>> But it only computes the first part (source).
>>>
>>> If I have 3 items without a source and 2 others with no author, then I
>>> should get 5 results instead only the first 3. What am I doing wrong?
>>>
>>> Thanks in advance,
>>> Katja
>>>
>>>
>>>
>>
>
> --
> Dipl.-Medieninf. Katja Siegemund
> Research Assistant
>
> Technische Universität Dresden
> Department of Computer Science
>
> Phone +49 (0)40 572 44 55 1
> Email Katja.Siegemund@tu-dresden.de
> WWW   http://st.inf.tu-dresden.de/
>
>


-- 
ufficio:  0542 32640
mail:     fpanico@imolinfo.it
cell:      349 3508348
skype:  francies82@gmail.com

Re: using "or" in sparql

Posted by Katja Siegemund <Ka...@tu-dresden.de>.
Oh that's good! Thanks, I'll try this out. Especially "minus" seems more 
convenient :)

Am 10.09.2012 13:53, schrieb Andy Seaborne:
> On 10/09/12 12:16, Magnus Stuhr wrote:
>> Hi,
>>
>> I just mocked up a template really quick, but something like this 
>> should work:
>>
>> SELECT ?s WHERE {
>> {
>> { ?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
>> UNION
>> {?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}}
>> UNION
>> {?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}}
>> }
>> UNION
>> {?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
>> }
>>
>> On top of this you can add even more UNION clauses. Obviously, you 
>> have to replace the triple-patterns and solution variables with 
>> whatever you want to find in your query.
>>
>> Regards,
>> Magnus Stuhr
>
> Alternatives which are similar:
>
> 1/
>
> SELECT ?r
>    {?r a ro:FR.
>     ?r ?p ?a
>     FILTER ( ?p NOT IN (ro:hasSource, ro:isAuthoredBy) )
> }
>
> This has limited applicability -- it's using the fact you have only 
> one triple pattern in the NOT EXISTS clause.
>
> 2/
>
> SELECT ?r
> { ?r a ro:FR.
>     MINUS { ?r ro:hasSource ?a }
>     MINUS { ?r ro:isAuthoredBy ?a }
> }
>
> Which style is best depends on how you think about the query and the 
> shape of the data.
>
>     Andy
>
>
>
>>
>> -----Original Message-----
>> From: Katja Siegemund [mailto:Katja.Siegemund@tu-dresden.de]
>> Sent: 10. september 2012 13:03
>> To: users@jena.apache.org
>> Subject: Re: using "or" in sparql
>>
>> Hm, well, but how can I use UNION for multiple times? I have a lot of 
>> "or's" in my query :(
>>
>>
>> Am 10.09.2012 12:29, schrieb Francesco Panico:
>>> Hi
>>>
>>> Your query selects all triples without  "source", at first. If  the two
>>> elements without  an "author" have a "source", they will be discarded,
>>> because the values added by "optional" clause have to respect "where
>>> clauses" anyway.
>>> You can check it.
>>>
>>> You can try to use UNION construct, because you want to retrieve a list
>>> with
>>> -item without a source
>>> -item without author
>>>
>>> SELECT ?r
>>> WHERE {?r a ro:FR.
>>>                    FILTER NOT EXISTS {?r ro:hasSource ?a}
>>>
>>> UNION{    ?r a ro:FR.
>>>                     FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>>>               }
>>> }
>>>
>>>
>>> I hope it is useful.
>>> Regards
>>> Francesco.
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>
>>>
>>>> Hi,
>>>>
>>>> I'm having trouble with constructing a query that uses "or". I already
>>>> tried optional, but it does not lead to the expected result. I'd be 
>>>> glad if
>>>> someone could help me. I'd like to query for:
>>>>
>>>> all var bindings that have no author or no source.
>>>>
>>>> I tried:
>>>>
>>>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>>>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>>>
>>>> But it only computes the first part (source).
>>>>
>>>> If I have 3 items without a source and 2 others with no author, then I
>>>> should get 5 results instead only the first 3. What am I doing wrong?
>>>>
>>>> Thanks in advance,
>>>> Katja
>>>>
>>>>
>>>
>>
>>
>


-- 
Dipl.-Medieninf. Katja Siegemund
Research Assistant

Technische Universität Dresden
Department of Computer Science

Phone +49 (0)40 572 44 55 1
Email Katja.Siegemund@tu-dresden.de
WWW   http://st.inf.tu-dresden.de/


Re: using "or" in sparql

Posted by Andy Seaborne <an...@apache.org>.
On 10/09/12 12:16, Magnus Stuhr wrote:
> Hi,
>
> I just mocked up a template really quick, but something like this should work:
>
> SELECT ?s WHERE {
> {
> { ?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
> UNION
> {?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}}
> UNION
> {?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}}
> }
> UNION
> {?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
> }
>
> On top of this you can add even more UNION clauses. Obviously, you have to replace the triple-patterns and solution variables with whatever you want to find in your query.
>
> Regards,
> Magnus Stuhr

Alternatives which are similar:

1/

SELECT ?r
    {?r a ro:FR.
     ?r ?p ?a
     FILTER ( ?p NOT IN (ro:hasSource, ro:isAuthoredBy) )
}

This has limited applicability -- it's using the fact you have only one 
triple pattern in the NOT EXISTS clause.

2/

SELECT ?r
{ ?r a ro:FR.
     MINUS { ?r ro:hasSource ?a }
     MINUS { ?r ro:isAuthoredBy ?a }
}

Which style is best depends on how you think about the query and the 
shape of the data.

	Andy



>
> -----Original Message-----
> From: Katja Siegemund [mailto:Katja.Siegemund@tu-dresden.de]
> Sent: 10. september 2012 13:03
> To: users@jena.apache.org
> Subject: Re: using "or" in sparql
>
> Hm, well, but how can I use UNION for multiple times? I have a lot of "or's" in my query :(
>
>
> Am 10.09.2012 12:29, schrieb Francesco Panico:
>> Hi
>>
>> Your query selects all triples without  "source", at first. If  the two
>> elements without  an "author" have a "source", they will be discarded,
>> because the values added by "optional" clause have to respect "where
>> clauses" anyway.
>> You can check it.
>>
>> You can try to use UNION construct, because you want to retrieve a list
>> with
>> -item without a source
>> -item without author
>>
>> SELECT ?r
>> WHERE {?r a ro:FR.
>>                    FILTER NOT EXISTS {?r ro:hasSource ?a}
>>
>> UNION{    ?r a ro:FR.
>>                     FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>>               }
>> }
>>
>>
>> I hope it is useful.
>> Regards
>> Francesco.
>>
>>
>>
>>
>>
>>
>> 2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>
>>
>>> Hi,
>>>
>>> I'm having trouble with constructing a query that uses "or". I already
>>> tried optional, but it does not lead to the expected result. I'd be glad if
>>> someone could help me. I'd like to query for:
>>>
>>> all var bindings that have no author or no source.
>>>
>>> I tried:
>>>
>>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>>
>>> But it only computes the first part (source).
>>>
>>> If I have 3 items without a source and 2 others with no author, then I
>>> should get 5 results instead only the first 3. What am I doing wrong?
>>>
>>> Thanks in advance,
>>> Katja
>>>
>>>
>>
>
>


RE: using "or" in sparql

Posted by Magnus Stuhr <Ma...@computas.com>.
Hi,

I just mocked up a template really quick, but something like this should work:

SELECT ?s WHERE {
{
{ ?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
UNION 
{?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}} 
UNION 
{?s ?p ?o.  FILTER NOT EXISTS {?s ?p ?o}}
}
UNION 
{?s ?p ?o. FILTER NOT EXISTS {?s ?p ?o}}
}

On top of this you can add even more UNION clauses. Obviously, you have to replace the triple-patterns and solution variables with whatever you want to find in your query.

Regards,
Magnus Stuhr

-----Original Message-----
From: Katja Siegemund [mailto:Katja.Siegemund@tu-dresden.de] 
Sent: 10. september 2012 13:03
To: users@jena.apache.org
Subject: Re: using "or" in sparql

Hm, well, but how can I use UNION for multiple times? I have a lot of "or's" in my query :(


Am 10.09.2012 12:29, schrieb Francesco Panico:
> Hi
>
> Your query selects all triples without  "source", at first. If  the two
> elements without  an "author" have a "source", they will be discarded,
> because the values added by "optional" clause have to respect "where
> clauses" anyway.
> You can check it.
>
> You can try to use UNION construct, because you want to retrieve a list
> with
> -item without a source
> -item without author
>
> SELECT ?r
> WHERE {?r a ro:FR.
>                   FILTER NOT EXISTS {?r ro:hasSource ?a}
>
> UNION{    ?r a ro:FR.
>                    FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>              }
> }
>
>
> I hope it is useful.
> Regards
> Francesco.
>
>
>
>
>
>
> 2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>
>
>> Hi,
>>
>> I'm having trouble with constructing a query that uses "or". I already
>> tried optional, but it does not lead to the expected result. I'd be glad if
>> someone could help me. I'd like to query for:
>>
>> all var bindings that have no author or no source.
>>
>> I tried:
>>
>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>
>> But it only computes the first part (source).
>>
>> If I have 3 items without a source and 2 others with no author, then I
>> should get 5 results instead only the first 3. What am I doing wrong?
>>
>> Thanks in advance,
>> Katja
>>
>>
>


-- 
Dipl.-Medieninf. Katja Siegemund
Research Assistant

Technische Universität Dresden
Department of Computer Science

Phone +49 (0)40 572 44 55 1
Email Katja.Siegemund@tu-dresden.de
WWW   http://st.inf.tu-dresden.de/


Re: using "or" in sparql

Posted by Katja Siegemund <Ka...@tu-dresden.de>.
Hm, well, but how can I use UNION for multiple times? I have a lot of 
"or's" in my query :(


Am 10.09.2012 12:29, schrieb Francesco Panico:
> Hi
>
> Your query selects all triples without  "source", at first. If  the two
> elements without  an "author" have a "source", they will be discarded,
> because the values added by "optional" clause have to respect "where
> clauses" anyway.
> You can check it.
>
> You can try to use UNION construct, because you want to retrieve a list
> with
> -item without a source
> -item without author
>
> SELECT ?r
> WHERE {?r a ro:FR.
>                   FILTER NOT EXISTS {?r ro:hasSource ?a}
>
> UNION{    ?r a ro:FR.
>                    FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
>              }
> }
>
>
> I hope it is useful.
> Regards
> Francesco.
>
>
>
>
>
>
> 2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>
>
>> Hi,
>>
>> I'm having trouble with constructing a query that uses "or". I already
>> tried optional, but it does not lead to the expected result. I'd be glad if
>> someone could help me. I'd like to query for:
>>
>> all var bindings that have no author or no source.
>>
>> I tried:
>>
>> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
>> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>>
>> But it only computes the first part (source).
>>
>> If I have 3 items without a source and 2 others with no author, then I
>> should get 5 results instead only the first 3. What am I doing wrong?
>>
>> Thanks in advance,
>> Katja
>>
>>
>


-- 
Dipl.-Medieninf. Katja Siegemund
Research Assistant

Technische Universität Dresden
Department of Computer Science

Phone +49 (0)40 572 44 55 1
Email Katja.Siegemund@tu-dresden.de
WWW   http://st.inf.tu-dresden.de/


Re: using "or" in sparql

Posted by Francesco Panico <fp...@imolinfo.it>.
Hi

Your query selects all triples without  "source", at first. If  the two
elements without  an "author" have a "source", they will be discarded,
because the values added by "optional" clause have to respect "where
clauses" anyway.
You can check it.

You can try to use UNION construct, because you want to retrieve a list
with
-item without a source
-item without author

SELECT ?r
WHERE {?r a ro:FR.
                 FILTER NOT EXISTS {?r ro:hasSource ?a}

UNION{    ?r a ro:FR.
                  FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}
            }
}


I hope it is useful.
Regards
Francesco.






2012/9/10 Katja Siegemund <Ka...@tu-dresden.de>

> Hi,
>
> I'm having trouble with constructing a query that uses "or". I already
> tried optional, but it does not lead to the expected result. I'd be glad if
> someone could help me. I'd like to query for:
>
> all var bindings that have no author or no source.
>
> I tried:
>
> SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a}
> OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";
>
> But it only computes the first part (source).
>
> If I have 3 items without a source and 2 others with no author, then I
> should get 5 results instead only the first 3. What am I doing wrong?
>
> Thanks in advance,
> Katja
>
>


-- 
ufficio:  0542 32640
mail:     fpanico@imolinfo.it
cell:      349 3508348
skype:  francies82@gmail.com

using "or" in sparql

Posted by Katja Siegemund <Ka...@tu-dresden.de>.
Hi,

I'm having trouble with constructing a query that uses "or". I already 
tried optional, but it does not lead to the expected result. I'd be glad 
if someone could help me. I'd like to query for:

all var bindings that have no author or no source.

I tried:

SELECT ?r WHERE {?r a ro:FR. FILTER NOT EXISTS {?r ro:hasSource ?a} 
OPTIONAL {FILTER NOT EXISTS {?r ro:isAuthoredBy ?a}}}";

But it only computes the first part (source).

If I have 3 items without a source and 2 others with no author, then I 
should get 5 results instead only the first 3. What am I doing wrong?

Thanks in advance,
Katja


RE: SDB and concurrent update

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Andy,

Thanks for the reply.

As a response to your answer on #1, note that the store descriptor file (quoted in the message) contains no such file string as "c:\X\cornerstone-lte2\config\planStore.ttl". That was the configFile argument to connectStore.

The Javadoc for SDBFactory.connectStore(String configFile) says: "configFile - Filename for assembler for Store and SDBConenction". It should either make explicit that it expects a URI or convert the filename to a URI internally.

Dave

-----Original Message-----
From: Andy Seaborne [mailto:andy.seaborne.apache@gmail.com] On Behalf Of Andy Seaborne
Sent: Monday, September 10, 2012 4:27 PM
To: users@jena.apache.org
Subject: Re: SDB and concurrent update

> 1. warning about .ttl file contents. I compared mine against the one in the Jena 2.7.3 distro, including byte-for-byte, and I see nothing that explains this. It is, however, "only" a warning.
>
>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 6, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#store> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.

Two things are bad here:

c:
\

although it's only the \ that is actually causing the warning.

A URI like:

<c:/X/c...>

parses as a URI scheme called "c:" (c.f. http:) which isn't what you mean.  The correct [*] URI is file:///c:/.  yes - 3 /// - there is an empty host betwen the first two / and the third.

file://host/c:/

\ is not allowed in URIs.  try / (good news - MS Windoes accepts / in file names)

[*] for some value of correct as the "file:" RFC is a bit hazy in places.

>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 8, col: 24] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 10, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
>
> --- planStore.ttl ---
> @prefix sdb:     <http://jena.hpl.hp.com/2007/sdb#> .
> @prefix rdfs:	 <http://www.w3.org/2000/01/rdf-schema#> .
> @prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
> @prefix ja:      <http://jena.hpl.hp.com/2005/11/Assembler#> .
>
> <#store> rdf:type sdb:Store ;
>      sdb:layout         "layout2" ;
>      sdb:connection     <#conn> ;
> .
> <#conn> rdf:type sdb:SDBConnection ;
>      sdb:sdbType        "postgresql" ;
>      sdb:sdbHost        "localhost" ;
>      sdb:sdbName        "jenaStore" ;
>      .
> ---


Re: SDB and concurrent update

Posted by Andy Seaborne <an...@apache.org>.
On 13/09/12 21:28, Lebling, David (US SSA) wrote:
> Andy,
>
> While working on generating my own FileManager, I tried to print the LocatorFile in the default FileManager (FileManager.get()). Its Object.hashCode() implementation doesn't check for there not being an altDir, which apparently is the case for the default instance. Result: NullPointerException, LocatorFile.java, line 95.
>
> Dave
>
> -----Original Message-----
> From: Andy Seaborne [mailto:andy.seaborne.apache@gmail.com] On Behalf Of Andy Seaborne
> Sent: Monday, September 10, 2012 4:27 PM
>
>> 3. A request, rather than a comment. During my debugging, one thing
>> Iwas seeing is a long, long pause while Jena goes out to the web after
>> unsuccessfully looking for a local ontology cached in the
>> OntDocumentManager (via loadImport). It would have helped during
>> debugging this if I knew how to set either the OntDocumentManager or
>> the FileManager to say "all my ontologies are cached in the
>> OntDocumentManager, so if you see yourself going out to the web, throw
>> an exception immediately, rather than even trying to connect." The
>> users of this software will be in an environment where every ontology
>> used is locally cached.
>
> Is this from code?  You can build your own FileManager and not include the URL locator, then it will not go to the web (c.f. Fuseki protects local files by creating a FileManger that does not access local files).
>
> 	Andy
>

Fixed, thanks.

A quick note: this area is due for change - there is a new I/O framework 
with (minor) changes to FileManger and friends [1].  Nothing 
architectural but there may be a few noticeable changes.

e.g. TypeStream becomes TypedInputStream since we also want a 
TypedOutputStream as well.  Other than the name change, it won't affect 
code (the current one does not work!).

	Andy

[1] https://svn.apache.org/repos/asf/jena/Experimental/riot-reader/


RE: SDB and concurrent update

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Andy,

While working on generating my own FileManager, I tried to print the LocatorFile in the default FileManager (FileManager.get()). Its Object.hashCode() implementation doesn't check for there not being an altDir, which apparently is the case for the default instance. Result: NullPointerException, LocatorFile.java, line 95.

Dave

-----Original Message-----
From: Andy Seaborne [mailto:andy.seaborne.apache@gmail.com] On Behalf Of Andy Seaborne
Sent: Monday, September 10, 2012 4:27 PM

> 3. A request, rather than a comment. During my debugging, one thing 
> Iwas seeing is a long, long pause while Jena goes out to the web after 
> unsuccessfully looking for a local ontology cached in the 
> OntDocumentManager (via loadImport). It would have helped during 
> debugging this if I knew how to set either the OntDocumentManager or 
> the FileManager to say "all my ontologies are cached in the 
> OntDocumentManager, so if you see yourself going out to the web, throw 
> an exception immediately, rather than even trying to connect." The 
> users of this software will be in an environment where every ontology 
> used is locally cached.

Is this from code?  You can build your own FileManager and not include the URL locator, then it will not go to the web (c.f. Fuseki protects local files by creating a FileManger that does not access local files).

	Andy


Re: SDB and concurrent update

Posted by Andy Seaborne <an...@apache.org>.
On 10/09/12 20:40, Lebling, David (US SSA) wrote:
> Andy Seabourne suggested I try the SDB 1.3.5 release candidate once JENA-118 was identified as a likely culprit.
>
> Here are my findings so far. The tl;dr version is that JENA-118 seems to be fixed but I still have a few comments and questions.

Great ... thanks for the response

>
> 0. I had to update to the latest Jena release (2.7.3) from our old and dusty 2.6.4, and by the usual Java dependency cascade updated all the stuff in the Jena distro. The newer versions resulted in the following issues:
>
> 1. warning about .ttl file contents. I compared mine against the one in the Jena 2.7.3 distro, including byte-for-byte, and I see nothing that explains this. It is, however, "only" a warning.
>
>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 6, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#store> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.

Two things are bad here:

c:
\

although it's only the \ that is actually causing the warning.

A URI like:

<c:/X/c...>

parses as a URI scheme called "c:" (c.f. http:) which isn't what you 
mean.  The correct [*] URI is file:///c:/.  yes - 3 /// - there is an 
empty host betwen the first two / and the third.

file://host/c:/

\ is not allowed in URIs.  try / (good news - MS Windoes accepts / in 
file names)

[*] for some value of correct as the "file:" RFC is a bit hazy in places.

>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 8, col: 24] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
>   WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 10, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
>
> --- planStore.ttl ---
> @prefix sdb:     <http://jena.hpl.hp.com/2007/sdb#> .
> @prefix rdfs:	 <http://www.w3.org/2000/01/rdf-schema#> .
> @prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
> @prefix ja:      <http://jena.hpl.hp.com/2005/11/Assembler#> .
>
> <#store> rdf:type sdb:Store ;
>      sdb:layout         "layout2" ;
>      sdb:connection     <#conn> ;
> .
> <#conn> rdf:type sdb:SDBConnection ;
>      sdb:sdbType        "postgresql" ;
>      sdb:sdbHost        "localhost" ;
>      sdb:sdbName        "jenaStore" ;
>      .
> ---
>
> 2.  SPARQL query it didn't like ("Queries with FROM/FROM NAMED not supported"). I realize the FROM UnionGraph (and really the GRAPH ?g) aren't necessary but should they be errors? I removed the extraneous/unsupported stuff and the query works.

"Queries with dataset descriptions (FROM/FROM NAMED) not supported" if 
its the error from SDB that I think it is.

In Fuseki, now passed down any FROM/FROM NAMED in the query - it works 
with dynamic datasets in TDB.

But SDB does not support the feature.  The engine only supports queries 
on the fixed dataset, possibly with union graph set.

>
> ---
> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
> SELECT ?s
> FROM <urn:x-arq:UnionGraph>
> WHERE {
>    { GRAPH ?g { ?s rdf:type <http://anchor/xxx> }  }
>    UNION
>    { GRAPH ?g { ?s rdf:type <http://anchor/yyy> }  }
>    UNION
>    { GRAPH ?g { ?s rdf:type <http://anchor/zzz> }  }  }
> ---
>
> 3. A request, rather than a comment. During my debugging, one thing
> Iwas seeing is a long, long pause while Jena goes out to the web after
> unsuccessfully looking for a local ontology cached in the
> OntDocumentManager (via loadImport). It would have helped during
> debugging this if I knew how to set either the OntDocumentManager or the
> FileManager to say "all my ontologies are cached in the
> OntDocumentManager, so if you see yourself going out to the web, throw
> an exception immediately, rather than even trying to connect." The users
> of this software will be in an environment where every ontology used is
> locally cached.

Is this from code?  You can build your own FileManager and not include 
the URL locator, then it will not go to the web (c.f. Fuseki protects 
local files by creating a FileManger that does not access local files).

	Andy

>
> Thanks to Andy and everyone else; you have all been extremely helpful.
>
> Dave
>


RE: SDB and concurrent update

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Andy Seabourne suggested I try the SDB 1.3.5 release candidate once JENA-118 was identified as a likely culprit.

Here are my findings so far. The tl;dr version is that JENA-118 seems to be fixed but I still have a few comments and questions. 

0. I had to update to the latest Jena release (2.7.3) from our old and dusty 2.6.4, and by the usual Java dependency cascade updated all the stuff in the Jena distro. The newer versions resulted in the following issues:

1. warning about .ttl file contents. I compared mine against the one in the Jena 2.7.3 distro, including byte-for-byte, and I see nothing that explains this. It is, however, "only" a warning.

 WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 6, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#store> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
 WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 8, col: 24] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
 WARN [tomcat-http--18] (ErrorHandlerFactory.java:68) - [line: 10, col: 1 ] Bad IRI: <c:\X\cornerstone-lte2\config\planStore.ttl#conn> Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.

--- planStore.ttl ---
@prefix sdb:     <http://jena.hpl.hp.com/2007/sdb#> .
@prefix rdfs:	 <http://www.w3.org/2000/01/rdf-schema#> .
@prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ja:      <http://jena.hpl.hp.com/2005/11/Assembler#> .

<#store> rdf:type sdb:Store ;
    sdb:layout         "layout2" ;
    sdb:connection     <#conn> ;
.
<#conn> rdf:type sdb:SDBConnection ;
    sdb:sdbType        "postgresql" ;
    sdb:sdbHost        "localhost" ;
    sdb:sdbName        "jenaStore" ;
    .
---

2.  SPARQL query it didn't like ("Queries with FROM/FROM NAMED not supported"). I realize the FROM UnionGraph (and really the GRAPH ?g) aren't necessary but should they be errors? I removed the extraneous/unsupported stuff and the query works.

---
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?s
FROM <urn:x-arq:UnionGraph>
WHERE {
  { GRAPH ?g { ?s rdf:type <http://anchor/xxx> }  }
  UNION
  { GRAPH ?g { ?s rdf:type <http://anchor/yyy> }  }
  UNION
  { GRAPH ?g { ?s rdf:type <http://anchor/zzz> }  }  }
---

3. A request, rather than a comment. During my debugging, one thing I was seeing is a long, long pause while Jena goes out to the web after unsuccessfully looking for a local ontology cached in the OntDocumentManager (via loadImport). It would have helped during debugging this if I knew how to set either the OntDocumentManager or the FileManager to say "all my ontologies are cached in the OntDocumentManager, so if you see yourself going out to the web, throw an exception immediately, rather than even trying to connect." The users of this software will be in an environment where every ontology used is locally cached.

Thanks to Andy and everyone else; you have all been extremely helpful.

Dave


Re: SDB and concurrent update

Posted by Andy Seaborne <an...@apache.org>.
On 09/09/12 21:05, Lebling, David (US SSA) wrote:
> Andy,
>
> Based on Joshua Taylor's advice, I added a connection pool for writes to ensure that any concurrent writes have different connections.
>
> I use transactions (model.begin()/model.commit()) for write, and using SDBFactory.connectStore() to get new connections each time the connection pool is empty.
>
> I am using SDB 1.3.4 and Postgres 9.0-801 (based on the mvn artifact versions).

Would it be possible to try the release candidate for 1.3.5?

	Andy

Repository:
https://repository.apache.org/content/repositories/snapshots/org/apache/jena/jena-sdb/ 


(email:
http://mail-archives.apache.org/mod_mbox/jena-users/201208.mbox/%3C503F8BBF.20306%40apache.org%3E
)

>
> Dave
>
> -----Original Message-----
> From: Andy Seaborne [mailto:andy.seaborne.apache@gmail.com] On Behalf Of Andy Seaborne
> Sent: Sunday, September 09, 2012 2:16 PM
> To: users@jena.apache.org
> Subject: Re: SDb and concurrent update
>
> Which version of SDB are you using? (And of PostgreSQL?)
>
> This might be JENA-118 [118].
>
> You will need multiple store connections (multiple JDBC connections) if you want concurrent transactions.  Locking is not connected internally to transactions.
>
> 	Andy
>


RE: SDb and concurrent update

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Andy,

Based on Joshua Taylor's advice, I added a connection pool for writes to ensure that any concurrent writes have different connections.

I use transactions (model.begin()/model.commit()) for write, and using SDBFactory.connectStore() to get new connections each time the connection pool is empty.

I am using SDB 1.3.4 and Postgres 9.0-801 (based on the mvn artifact versions).

Dave

-----Original Message-----
From: Andy Seaborne [mailto:andy.seaborne.apache@gmail.com] On Behalf Of Andy Seaborne
Sent: Sunday, September 09, 2012 2:16 PM
To: users@jena.apache.org
Subject: Re: SDb and concurrent update

Which version of SDB are you using? (And of PostgreSQL?)

This might be JENA-118 [118].

You will need multiple store connections (multiple JDBC connections) if you want concurrent transactions.  Locking is not connected internally to transactions.

	Andy


Re: SDb and concurrent update

Posted by Andy Seaborne <an...@apache.org>.
On 09/09/12 17:55, Lebling, David (US SSA) wrote:
> Joshua,
>
> I implemented a simple connection pool. Each client that wishes to write to the SDB database asks the pool for a connection. If one is available it is used, otherwise a new connection is made. When the writer is done with the connection it is returned to the pool.
>
> This has definitely improved the behavior. However, I find that I am still getting SQLExceptions on some write operations. In all cases these exceptions occur in a thread I don't recognize: "Thread-18" or "Thread-19", running the "tuple loader."
>
> An example:
>
> ------
> WARN [Thread-18] (SDBConnection.java:326) - execUpdate: SQLException
> ERROR: duplicate key value violates unique constraint "nodes_pkey"
>    Detail: Key (hash)=(3239162530368006953) already exists.
> INSERT INTO Nodes (hash, lex, lang, datatype, type)
> SELECT NNodeQuads1899993940.n0 , NNodeQuads1899993940.n1 , NNodeQuads1899993940.n2 , NNodeQuads1899993940.n3 , NNodeQuads1899993940.n4
> FROM NNodeQuads1899993940 LEFT JOIN Nodes ON (NNodeQuads1899993940.n0=Nodes.hash)
> WHERE Nodes.hash IS NULL
>
> ERROR [Thread-18] (LoaderTuplesNodes.java:345) - Error in thread: Exception flushing
> com.hp.hpl.jena.sdb.SDBException: Exception flushing
> 	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.flush(TupleLoaderBase.java:220)
> 	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.finish(TupleLoaderBase.java:155)
> 	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.commitTuples(LoaderTuplesNodes.java:283)
> 	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.access$100(LoaderTuplesNodes.java:31)
> 	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes$Commiter.run(LoaderTuplesNodes.java:318)
> 	at java.lang.Thread.run(Thread.java:662)
> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "nodes_pkey"
>    Detail: Key (hash)=(3239162530368006953) already exists.
> 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
> 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
> 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> 	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
> 	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
> 	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:302)
> 	at com.hp.hpl.jena.sdb.sql.SDBConnection.execUpdate(SDBConnection.java:151)
> 	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.flush(TupleLoaderBase.java:202)
> 	... 5 more
> ------
>
> Any ideas as to cause and cure?

Which version of SDB are you using? (And of PostgreSQL?)

This might be JENA-118 [118].

You will need multiple store connections (multiple JDBC connections) if 
you want concurrent transactions.  Locking is not connected internally 
to transactions.

	Andy

[118] https://issues.apache.org/jira/browse/JENA-118

>
> Dave
>
>
> -----Original Message-----
> From: Joshua TAYLOR [mailto:joshuaaaron@gmail.com]
> Sent: Friday, September 07, 2012 3:17 PM
> To: users@jena.apache.org
> Subject: Re: Model critical sections
>
> On Fri, Sep 7, 2012 at 2:39 PM, Lebling, David (US SSA) <da...@baesystems.com> wrote:
>> There is a single service (and JVM) that writes the models (at least in this case), and many that read them. There is a single Store (and hence SDBConnection) for the service.
>
> I'm sorry, I'm not sure I follow.  Let me mirror this back, and please correct me if I'm wrong.  There is a single JVM running a bunch of threads.  One of these threads writes, and the rest only read.  Since it's all within the same JVM (and there's only one store), there's only one SDBConnection.
>
>> I originally wrote the code so that it used the model.enterCriticalSection/model.leaveCriticalSection methods. This did not seem to prevent other threads from writing the same model inside the critical section.
>>
>> My belief is that model.begin/model.commit aren't permitted to nest, so it's key to avoid one thread trying to do a transaction when another already is.
>>
>> So, I tried doing both: model.enterCriticalSection + model.begin, finishing with model.commit + model.leaveCriticalSection. That doesn't seem to do the trick either. I'm still getting "already in transaction," meaning other threads are trying to write the same model and not being blocked by the critical section.
>>
>> So, is the ModelLock not a thread-safe lock?
>
> I'm not sure what the purpose of the lock would be if it weren't thread safe;  it's there to permit concurrent access to a model.  The model generation code
>
> Model model = SDBFactory.connectNamedModel(store, name);
>
> can produce different models on different invocation, and hence the Locks would apply to different models, but the transaction handler can be the same for them.  After a coworker ran a quick test, it looks like the following can happen.
>
> Model m1 = SDBFactory.connectNamedModel(store, name); Model m2 = SDBFactory.connectNamedModel(store, name);
>
> m1 and m2 are distinct models, so their enter/leaveCriticalSections aren't going to interfere; you'll be able to enter both critical sections at once.  However, they're built from the same underlying Store, and
>
> m1.getTransactionHandler() == m2.getTransactionHandler()
>
> so when you begin() with m1, and subsequently begin() with m2, you're getting the nesting that you're seeing.  So, you could either:
>
> 1) be sure to use the *same* model everywhere so that the enter/leaveCriticalSections do what you want; or
> 2) give each writing thread its own Store and connection, and don't use begin/commit in the reading threads.  You shouldn't need to use enter/leaveCriticalSection at all in this case. (The writers will be covered by begin/commit, and the readers don't have to worry about read locks since nothing will ever be writing to those models (the only model getting written to is the one that the writer has access
> to).)
>
> The first is easier if you'll only be using one JVM, but the second should scale to multiple JVMs.
>
> //JT
> --
> Joshua Taylor, http://www.cs.rpi.edu/~tayloj/
>


RE: Model critical sections

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Joshua,

I implemented a simple connection pool. Each client that wishes to write to the SDB database asks the pool for a connection. If one is available it is used, otherwise a new connection is made. When the writer is done with the connection it is returned to the pool.

This has definitely improved the behavior. However, I find that I am still getting SQLExceptions on some write operations. In all cases these exceptions occur in a thread I don't recognize: "Thread-18" or "Thread-19", running the "tuple loader."

An example:

------
WARN [Thread-18] (SDBConnection.java:326) - execUpdate: SQLException
ERROR: duplicate key value violates unique constraint "nodes_pkey"
  Detail: Key (hash)=(3239162530368006953) already exists.
INSERT INTO Nodes (hash, lex, lang, datatype, type) 
SELECT NNodeQuads1899993940.n0 , NNodeQuads1899993940.n1 , NNodeQuads1899993940.n2 , NNodeQuads1899993940.n3 , NNodeQuads1899993940.n4
FROM NNodeQuads1899993940 LEFT JOIN Nodes ON (NNodeQuads1899993940.n0=Nodes.hash) 
WHERE Nodes.hash IS NULL

ERROR [Thread-18] (LoaderTuplesNodes.java:345) - Error in thread: Exception flushing
com.hp.hpl.jena.sdb.SDBException: Exception flushing
	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.flush(TupleLoaderBase.java:220)
	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.finish(TupleLoaderBase.java:155)
	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.commitTuples(LoaderTuplesNodes.java:283)
	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes.access$100(LoaderTuplesNodes.java:31)
	at com.hp.hpl.jena.sdb.layout2.LoaderTuplesNodes$Commiter.run(LoaderTuplesNodes.java:318)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "nodes_pkey"
  Detail: Key (hash)=(3239162530368006953) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:302)
	at com.hp.hpl.jena.sdb.sql.SDBConnection.execUpdate(SDBConnection.java:151)
	at com.hp.hpl.jena.sdb.layout2.TupleLoaderBase.flush(TupleLoaderBase.java:202)
	... 5 more
------

Any ideas as to cause and cure?

Dave


-----Original Message-----
From: Joshua TAYLOR [mailto:joshuaaaron@gmail.com] 
Sent: Friday, September 07, 2012 3:17 PM
To: users@jena.apache.org
Subject: Re: Model critical sections

On Fri, Sep 7, 2012 at 2:39 PM, Lebling, David (US SSA) <da...@baesystems.com> wrote:
> There is a single service (and JVM) that writes the models (at least in this case), and many that read them. There is a single Store (and hence SDBConnection) for the service.

I'm sorry, I'm not sure I follow.  Let me mirror this back, and please correct me if I'm wrong.  There is a single JVM running a bunch of threads.  One of these threads writes, and the rest only read.  Since it's all within the same JVM (and there's only one store), there's only one SDBConnection.

> I originally wrote the code so that it used the model.enterCriticalSection/model.leaveCriticalSection methods. This did not seem to prevent other threads from writing the same model inside the critical section.
>
> My belief is that model.begin/model.commit aren't permitted to nest, so it's key to avoid one thread trying to do a transaction when another already is.
>
> So, I tried doing both: model.enterCriticalSection + model.begin, finishing with model.commit + model.leaveCriticalSection. That doesn't seem to do the trick either. I'm still getting "already in transaction," meaning other threads are trying to write the same model and not being blocked by the critical section.
>
> So, is the ModelLock not a thread-safe lock?

I'm not sure what the purpose of the lock would be if it weren't thread safe;  it's there to permit concurrent access to a model.  The model generation code

Model model = SDBFactory.connectNamedModel(store, name);

can produce different models on different invocation, and hence the Locks would apply to different models, but the transaction handler can be the same for them.  After a coworker ran a quick test, it looks like the following can happen.

Model m1 = SDBFactory.connectNamedModel(store, name); Model m2 = SDBFactory.connectNamedModel(store, name);

m1 and m2 are distinct models, so their enter/leaveCriticalSections aren't going to interfere; you'll be able to enter both critical sections at once.  However, they're built from the same underlying Store, and

m1.getTransactionHandler() == m2.getTransactionHandler()

so when you begin() with m1, and subsequently begin() with m2, you're getting the nesting that you're seeing.  So, you could either:

1) be sure to use the *same* model everywhere so that the enter/leaveCriticalSections do what you want; or
2) give each writing thread its own Store and connection, and don't use begin/commit in the reading threads.  You shouldn't need to use enter/leaveCriticalSection at all in this case. (The writers will be covered by begin/commit, and the readers don't have to worry about read locks since nothing will ever be writing to those models (the only model getting written to is the one that the writer has access
to).)

The first is easier if you'll only be using one JVM, but the second should scale to multiple JVMs.

//JT
--
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Re: Model critical sections

Posted by Joshua TAYLOR <jo...@gmail.com>.
On Fri, Sep 7, 2012 at 2:39 PM, Lebling, David (US SSA)
<da...@baesystems.com> wrote:
> There is a single service (and JVM) that writes the models (at least in this case), and many that read them. There is a single Store (and hence SDBConnection) for the service.

I'm sorry, I'm not sure I follow.  Let me mirror this back, and please
correct me if I'm wrong.  There is a single JVM running a bunch of
threads.  One of these threads writes, and the rest only read.  Since
it's all within the same JVM (and there's only one store), there's
only one SDBConnection.

> I originally wrote the code so that it used the model.enterCriticalSection/model.leaveCriticalSection methods. This did not seem to prevent other threads from writing the same model inside the critical section.
>
> My belief is that model.begin/model.commit aren't permitted to nest, so it's key to avoid one thread trying to do a transaction when another already is.
>
> So, I tried doing both: model.enterCriticalSection + model.begin, finishing with model.commit + model.leaveCriticalSection. That doesn't seem to do the trick either. I'm still getting "already in transaction," meaning other threads are trying to write the same model and not being blocked by the critical section.
>
> So, is the ModelLock not a thread-safe lock?

I'm not sure what the purpose of the lock would be if it weren't
thread safe;  it's there to permit concurrent access to a model.  The
model generation code

Model model = SDBFactory.connectNamedModel(store, name);

can produce different models on different invocation, and hence the
Locks would apply to different models, but the transaction handler can
be the same for them.  After a coworker ran a quick test, it looks
like the following can happen.

Model m1 = SDBFactory.connectNamedModel(store, name);
Model m2 = SDBFactory.connectNamedModel(store, name);

m1 and m2 are distinct models, so their enter/leaveCriticalSections
aren't going to interfere; you'll be able to enter both critical
sections at once.  However, they're built from the same underlying
Store, and

m1.getTransactionHandler() == m2.getTransactionHandler()

so when you begin() with m1, and subsequently begin() with m2, you're
getting the nesting that you're seeing.  So, you could either:

1) be sure to use the *same* model everywhere so that the
enter/leaveCriticalSections do what you want; or
2) give each writing thread its own Store and connection, and don't
use begin/commit in the reading threads.  You shouldn't need to use
enter/leaveCriticalSection at all in this case. (The writers will be
covered by begin/commit, and the readers don't have to worry about
read locks since nothing will ever be writing to those models (the
only model getting written to is the one that the writer has access
to).)

The first is easier if you'll only be using one JVM, but the second
should scale to multiple JVMs.

//JT
-- 
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

RE: Model critical sections

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
There is a single service (and JVM) that writes the models (at least in this case), and many that read them. There is a single Store (and hence SDBConnection) for the service.

I originally wrote the code so that it used the model.enterCriticalSection/model.leaveCriticalSection methods. This did not seem to prevent other threads from writing the same model inside the critical section.

My belief is that model.begin/model.commit aren't permitted to nest, so it's key to avoid one thread trying to do a transaction when another already is.

So, I tried doing both: model.enterCriticalSection + model.begin, finishing with model.commit + model.leaveCriticalSection. That doesn't seem to do the trick either. I'm still getting "already in transaction," meaning other threads are trying to write the same model and not being blocked by the critical section.

So, is the ModelLock not a thread-safe lock?

Dave

-----Original Message-----
From: Joshua TAYLOR [mailto:joshuaaaron@gmail.com] 
Sent: Friday, September 07, 2012 1:09 PM
To: users@jena.apache.org
Subject: Re: Model critical sections

On Fri, Sep 7, 2012 at 12:17 PM, Lebling, David (US SSA) <da...@baesystems.com> wrote:
> Thanks for the suggestion. I added a TransactionHandlerSDB to my code that writes to the repository, and appropriate begin()/commit() pairs.
>
> I have multiple clients calling the code fairly often (this is a stress test case, by the way), and what happens is that one client's input starts a write transaction (begin()) on a graph and before it finishes (commit()) another client's input starts another write transaction on the same graph. An exception "Already in a transaction" is thrown. Does that mean that begin() doesn't block if a transaction is in progress? (The code just seems to check a flag before throwing the exception.) Does the user of a TransactionHandler have to perform his own blocking? There doesn't seem to be a way to check if a transaction is in progress before calling begin().

(Having discussed this with a coworker...) Are you getting these issues within a single JVM with multiple threads that all use the same JDBC connection?  If this is the case, then this thread from 2010 [1] is relevant, which states, "If you need transactions between the concurrent operations, you will either need to use multiple JDBC connections, or use a single connection and use in-JVM locking (e.g.
MRSW locks)."

> When I use this mechanism to do read transactions (i.e., to mimic the critical section behavior I had before), the commit() method says "WARN: nothing to commit".

For the reading, I don't think the transactions are necessary;  each read will be from whatever the last state of the database is.  Without transactions on the writes, you could end up reading from a model when only part of a write has happened.  With transactions on the writes, you'll only be reading from models where writes have been completed.
You could, of course, still be reading from a blank model if your read executes before any of your write transactions have been committed.

//JT

[1] http://tech.groups.yahoo.com/group/jena-dev/message/43942

--
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Re: Model critical sections

Posted by Joshua TAYLOR <jo...@gmail.com>.
On Fri, Sep 7, 2012 at 12:17 PM, Lebling, David (US SSA)
<da...@baesystems.com> wrote:
> Thanks for the suggestion. I added a TransactionHandlerSDB to my code that writes to the repository, and appropriate begin()/commit() pairs.
>
> I have multiple clients calling the code fairly often (this is a stress test case, by the way), and what happens is that one client's input starts a write transaction (begin()) on a graph and before it finishes (commit()) another client's input starts another write transaction on the same graph. An exception "Already in a transaction" is thrown. Does that mean that begin() doesn't block if a transaction is in progress? (The code just seems to check a flag before throwing the exception.) Does the user of a TransactionHandler have to perform his own blocking? There doesn't seem to be a way to check if a transaction is in progress before calling begin().

(Having discussed this with a coworker...) Are you getting these
issues within a single JVM with multiple threads that all use the same
JDBC connection?  If this is the case, then this thread from 2010 [1]
is relevant, which states, "If you need transactions between the
concurrent operations, you will either need to use multiple JDBC
connections, or use a single connection and use in-JVM locking (e.g.
MRSW locks)."

> When I use this mechanism to do read transactions (i.e., to mimic the critical section behavior I had before), the commit() method says "WARN: nothing to commit".

For the reading, I don't think the transactions are necessary;  each
read will be from whatever the last state of the database is.  Without
transactions on the writes, you could end up reading from a model when
only part of a write has happened.  With transactions on the writes,
you'll only be reading from models where writes have been completed.
You could, of course, still be reading from a blank model if your read
executes before any of your write transactions have been committed.

//JT

[1] http://tech.groups.yahoo.com/group/jena-dev/message/43942

-- 
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

RE: Model critical sections

Posted by "Lebling, David (US SSA)" <da...@baesystems.com>.
Joshua,

Thanks for the suggestion. I added a TransactionHandlerSDB to my code that writes to the repository, and appropriate begin()/commit() pairs.

I have multiple clients calling the code fairly often (this is a stress test case, by the way), and what happens is that one client's input starts a write transaction (begin()) on a graph and before it finishes (commit()) another client's input starts another write transaction on the same graph. An exception "Already in a transaction" is thrown. Does that mean that begin() doesn't block if a transaction is in progress? (The code just seems to check a flag before throwing the exception.) Does the user of a TransactionHandler have to perform his own blocking? There doesn't seem to be a way to check if a transaction is in progress before calling begin().

When I use this mechanism to do read transactions (i.e., to mimic the critical section behavior I had before), the commit() method says "WARN: nothing to commit".

My top-level goal here is to be able to have multiple clients read and write named graphs in the SDB database without interference from other clients, just in case that's not clear.

Dave Lebling

-----Original Message-----
From: Joshua TAYLOR [mailto:joshuaaaron@gmail.com] 
Sent: Friday, September 07, 2012 10:41 AM
To: users@jena.apache.org
Subject: Re: Model critical sections

On Fri, Sep 7, 2012 at 10:34 AM, Joshua TAYLOR <jo...@gmail.com> wrote:
> On Fri, Sep 7, 2012 at 9:08 AM, Lebling, David (US SSA) 
> <da...@baesystems.com> wrote:
>> I have a suite of webservices which use OWL ontologies stored as named graphs in an SDB database. I also have a set of Java objects that provide a POJO façade over ontology class instances. These instances import many or even all of the OWL ontologies, so I have an OntDocumentManager which points to cached versions of the ontologies.
>>
>> When an ontology changes, I need to update it in the database, then re-cache it in the OntDocumentManager. During those operations (update ontology graph in database, update cached version in OntDocumentManager) I am within a try/finally block that (respectively) is a write lock or a read lock.
>>
>> So a sketch (not the full implementation) of the code for writing and reading would be:
>>
>> void update(String name, Model newModel) { Model model = 
>> SDBFactory.connectNamedModel(store, name); try {
>>                 model.enterCriticalSection(false); // write
>>                 model.removeAll();
>>                 model.add(newModel);
>> }
>> finally {
>>                 model.leaveCriticalSection(); } }
>>
>> void cache(OntDocumentManager mgr, String name) {
>>                 Model model = SDBFactory.connectNamedModel(store, name);
>>                 Model cachedModel = createModel(); // this method creates an OWL_MEM model with the doc mgr as its document manager, etc.
>>                 try {
>>                                 model.enterCriticalSection(true); // read
>>                                 cachedModel.add(model);
>>                 }
>>                 finally {
>>                                 model.leaveCriticalSection(); } 
>> mgr.addModel(name, cachedModel, true); }
>>
>> What appears to happen is that these critical sections don't appear to inhibit other users (in my tests I have five) of the SDB database from modifying the stored graph. Readers of the graphs sometimes end up with empty graphs. I will sometimes end up with stored graphs that are empty, or inconsistent in some way that eventually corrupts the underlying tables in the database, requiring a complete reinitialization. I also get IllegalThreadState exceptions. Many different exceptions are thrown, with little obvious consistency.
>>
>> Is critical section only good within a JVM? Do I need to use some sort of transaction instead?
>
> The "Concurrent access to models" document [1] describes the Lock 
> mechanisms that you're using now, but says that they're for the 
> "multiple threads in a single JVM" and to use transactions for 
> database backed models.  I haven't used SDB (or TDB), so I can't point 
> you to the documentation on those systems, but I think those are what 
> you need;  hopefully someone else will chime in.

After a bit more Googling, I think you might be interested in the javadoc pages for TransactionHandler [1] and TransactionHandlerSDB [2], and Graph#getTransactionHandler() [3].

Hope this helps,
//JT

[1] http://jena.apache.org/documentation/javadoc/jena/com/hp/hpl/jena/graph/TransactionHandler.html
[2] http://jena.apache.org/documentation/javadoc/sdb/com/hp/hpl/jena/sdb/graph/TransactionHandlerSDB.html
[3] http://jena.apache.org/documentation/javadoc/jena/com/hp/hpl/jena/graph/Graph.html#getTransactionHandler()
--
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Re: Model critical sections

Posted by Joshua TAYLOR <jo...@gmail.com>.
On Fri, Sep 7, 2012 at 10:34 AM, Joshua TAYLOR <jo...@gmail.com> wrote:
> On Fri, Sep 7, 2012 at 9:08 AM, Lebling, David (US SSA)
> <da...@baesystems.com> wrote:
>> I have a suite of webservices which use OWL ontologies stored as named graphs in an SDB database. I also have a set of Java objects that provide a POJO façade over ontology class instances. These instances import many or even all of the OWL ontologies, so I have an OntDocumentManager which points to cached versions of the ontologies.
>>
>> When an ontology changes, I need to update it in the database, then re-cache it in the OntDocumentManager. During those operations (update ontology graph in database, update cached version in OntDocumentManager) I am within a try/finally block that (respectively) is a write lock or a read lock.
>>
>> So a sketch (not the full implementation) of the code for writing and reading would be:
>>
>> void update(String name, Model newModel) {
>> Model model = SDBFactory.connectNamedModel(store, name);
>> try {
>>                 model.enterCriticalSection(false); // write
>>                 model.removeAll();
>>                 model.add(newModel);
>> }
>> finally {
>>                 model.leaveCriticalSection();
>> }
>> }
>>
>> void cache(OntDocumentManager mgr, String name) {
>>                 Model model = SDBFactory.connectNamedModel(store, name);
>>                 Model cachedModel = createModel(); // this method creates an OWL_MEM model with the doc mgr as its document manager, etc.
>>                 try {
>>                                 model.enterCriticalSection(true); // read
>>                                 cachedModel.add(model);
>>                 }
>>                 finally {
>>                                 model.leaveCriticalSection();
>> }
>> mgr.addModel(name, cachedModel, true);
>> }
>>
>> What appears to happen is that these critical sections don't appear to inhibit other users (in my tests I have five) of the SDB database from modifying the stored graph. Readers of the graphs sometimes end up with empty graphs. I will sometimes end up with stored graphs that are empty, or inconsistent in some way that eventually corrupts the underlying tables in the database, requiring a complete reinitialization. I also get IllegalThreadState exceptions. Many different exceptions are thrown, with little obvious consistency.
>>
>> Is critical section only good within a JVM? Do I need to use some sort of transaction instead?
>
> The "Concurrent access to models" document [1] describes the Lock
> mechanisms that you're using now, but says that they're for the
> "multiple threads in a single JVM" and to use transactions for
> database backed models.  I haven't used SDB (or TDB), so I can't point
> you to the documentation on those systems, but I think those are what
> you need;  hopefully someone else will chime in.

After a bit more Googling, I think you might be interested in the
javadoc pages for TransactionHandler [1] and TransactionHandlerSDB
[2], and Graph#getTransactionHandler() [3].

Hope this helps,
//JT

[1] http://jena.apache.org/documentation/javadoc/jena/com/hp/hpl/jena/graph/TransactionHandler.html
[2] http://jena.apache.org/documentation/javadoc/sdb/com/hp/hpl/jena/sdb/graph/TransactionHandlerSDB.html
[3] http://jena.apache.org/documentation/javadoc/jena/com/hp/hpl/jena/graph/Graph.html#getTransactionHandler()
-- 
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Re: Model critical sections

Posted by Joshua TAYLOR <jo...@gmail.com>.
On Fri, Sep 7, 2012 at 9:08 AM, Lebling, David (US SSA)
<da...@baesystems.com> wrote:
> I have a suite of webservices which use OWL ontologies stored as named graphs in an SDB database. I also have a set of Java objects that provide a POJO façade over ontology class instances. These instances import many or even all of the OWL ontologies, so I have an OntDocumentManager which points to cached versions of the ontologies.
>
> When an ontology changes, I need to update it in the database, then re-cache it in the OntDocumentManager. During those operations (update ontology graph in database, update cached version in OntDocumentManager) I am within a try/finally block that (respectively) is a write lock or a read lock.
>
> So a sketch (not the full implementation) of the code for writing and reading would be:
>
> void update(String name, Model newModel) {
> Model model = SDBFactory.connectNamedModel(store, name);
> try {
>                 model.enterCriticalSection(false); // write
>                 model.removeAll();
>                 model.add(newModel);
> }
> finally {
>                 model.leaveCriticalSection();
> }
> }
>
> void cache(OntDocumentManager mgr, String name) {
>                 Model model = SDBFactory.connectNamedModel(store, name);
>                 Model cachedModel = createModel(); // this method creates an OWL_MEM model with the doc mgr as its document manager, etc.
>                 try {
>                                 model.enterCriticalSection(true); // read
>                                 cachedModel.add(model);
>                 }
>                 finally {
>                                 model.leaveCriticalSection();
> }
> mgr.addModel(name, cachedModel, true);
> }
>
> What appears to happen is that these critical sections don't appear to inhibit other users (in my tests I have five) of the SDB database from modifying the stored graph. Readers of the graphs sometimes end up with empty graphs. I will sometimes end up with stored graphs that are empty, or inconsistent in some way that eventually corrupts the underlying tables in the database, requiring a complete reinitialization. I also get IllegalThreadState exceptions. Many different exceptions are thrown, with little obvious consistency.
>
> Is critical section only good within a JVM? Do I need to use some sort of transaction instead?

The "Concurrent access to models" document [1] describes the Lock
mechanisms that you're using now, but says that they're for the
"multiple threads in a single JVM" and to use transactions for
database backed models.  I haven't used SDB (or TDB), so I can't point
you to the documentation on those systems, but I think those are what
you need;  hopefully someone else will chime in.

//JT

[1] http://jena.apache.org/documentation/notes/concurrency-howto.html

-- 
Joshua Taylor, http://www.cs.rpi.edu/~tayloj/