You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Jean-Marc Vanel <je...@gmail.com> on 2017/11/15 11:18:14 UTC
Testing TBD2: first report: code changes to be made
Hi
I started to test TBD2 on my application semantic_forms .
I dumped the 3 databases with the TDB (1) tool,
then loaded with the TDB 2 tool :
runMain tdb2.tdbloader --loc TDB --verbose
/home/jmv/src/semantic_forms/scala/dump.nq
there where some bad quads with a literal subject from old bugs, that I had
to manually edit out , but not a big deal.
At runtime many things work, but I had one runtime error in /authenticate :
*[TransactionException: Iterator used outside its original transaction]*
Indeed , this is exactly that:
https://github.com/jmvanel/semantic_forms/blob/master/scala/forms/src/main/scala/deductions/runtime/services/Authentication.scala#L66
This was on my laptop.
I'll fix the code, which seems to be changes compatible with old TDB.
Then I'll upgrade to TDB2 on the sandbox, then on the real site.
Hope I don't bother you with non complete report .
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Jean-Marc Vanel <je...@gmail.com>.
OK, so currently no clear path to investigate .
Still willing to intrument the code, if you have some idea ...
I'll try to make the issue happen on my laptop in a reproductible way ...
2017-11-16 19:17 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>
> On 16/11/17 18:00, Jean-Marc Vanel wrote:
>
>>
>> /With TDB2, is it bad to share a Dataset across threads and transactions ?
>> What are the don't 's regarding TDB2 API use ?/
>>
>
> Got my (email) threads mixed up ... on another thread today:
>
> """
> TDB2 goals are to address the scale limitations on transactions, the
> write-back queue overload problems, a better architecture e.g. fully
> integrate in jena-text transactions, and no quirks about models across
> transactions.
> """
> http://jena.apache.org/documentation/tdb2/
>
> It is not bad.
>
> It *should* work - it's supposed to.
>
> As does (when some recent fixes go in): Graph.TransactionHandler.begin()
> including automatic promotion from R to W as required by that API.
>
> It's in the test suite for "across transaction, same thread" and I can't
> remember for multithread. The core engine for TDB2 was written a couple of
> years ago; I forget details.
>
> Transactions at the API level are per thread, internally they are not.
>
> Andy
>
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Andy Seaborne <an...@apache.org>.
On 16/11/17 18:00, Jean-Marc Vanel wrote:
>
> /With TDB2, is it bad to share a Dataset across threads and transactions ?
> What are the don't 's regarding TDB2 API use ?/
Got my (email) threads mixed up ... on another thread today:
"""
TDB2 goals are to address the scale limitations on transactions, the
write-back queue overload problems, a better architecture e.g. fully
integrate in jena-text transactions, and no quirks about models across
transactions.
"""
http://jena.apache.org/documentation/tdb2/
It is not bad.
It *should* work - it's supposed to.
As does (when some recent fixes go in): Graph.TransactionHandler.begin()
including automatic promotion from R to W as required by that API.
It's in the test suite for "across transaction, same thread" and I can't
remember for multithread. The core engine for TDB2 was written a couple
of years ago; I forget details.
Transactions at the API level are per thread, internally they are not.
Andy
Re: Testing TBD2: first report: code changes to be made
Posted by Jean-Marc Vanel <je...@gmail.com>.
Here is the complete thread dump attached .
An important question got lost in mail automatic shrinks:
*With TDB2, is it bad to share a Dataset across threads and transactions ?
What are the don't 's regarding TDB2 API use ?*
2017-11-16 18:33 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>
> On 16/11/17 17:17, Jean-Marc Vanel wrote:
>
>> Sorry that was not clear , rewriting.
>>
>
> Good - because was wondering who wrote what :-)
>
>
>> 2017-11-16 17:36 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>>
>>
>>> 2017-11-16 16:11 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>>
>>>
>>>> On 16/11/17 09:12, Jean-Marc Vanel wrote:
>>>>
>>>> So I tested on the sandbox site,
>>>>> and it got quickly frozen :( .
>>>>> ... <http://semantic-forms.cc:9111/>
>>>>>
>>>>>
>>>>> In shell server side, I did a kill -3 to get the threads dump,
>>>>> and there are 8 occurences of the same pattern in bold:
>>>>>
>>>>> "application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
>>>>> tid=0x00007fdb74007000 nid=0x5f5f waiting on condition
>>>>> [0x00007fdb881b8000]
>>>>> java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for <0x00000000f3f42378> (a
>>>>>
>>>>>
>>>>> java.util.concurrent.Semaphore$FairSync)
>>>>
>>>> Some other thread is holding the permit that control the number of
>>>> writers.
>>>>
>>>> The WriterLock is a Semaphore with one permit.
>>>>
>>>> The question becomes who has the permit and are they making progress
>>>>
>>>> Note - the lock is not re-entrant. begin-begin on the same thread will
>>>> deadlock (in case Akka is being clever about actors to threads).
>>>>
>>>>
>>> I assume from this that the error lies in my code, running fine with
>>> TDB1,
>>> but TDB2 is mode picky .
>>> I'm puzzled as to what to do.
>>> Which bad practices to look for in the code ?
>>> Instrumenting the code for run-time messages ?
>>>
>>> My understading is that Play! framework activates one thread from a pool
>>> to process the HTTP request, so here everything is sequential.
>>> I also start 2 or 3 futures, so each in another thread.
>>>
>>> So concurrent lock acquire can occur between threads, but I understood
>> that
>> it's allowed, and even more efficient in TDB2 because several writes can
>> occur concurrently.
>>
>
> TDB2 is like TDB1 single-active-writer.
>
> Embedded commits an single thread were always forbidden,
>> and I don't think that there are some, but this is possible.
>> When this happened in TDB1 , a TransactionException was thrown saying
>> "allready in transaction".
>> TDB2 does not throw in such case ?
>>
>
> I just checked it does.
>
> So who is holding the permit and why aren't they giving it up?
> Have they become blocked on something else ... which a waiting thread has?
> (deadlock by starvation).
>
>
>> It's not clear what in TDB 1 doc is still good , and what not :
>>> https://jena.apache.org/documentation/tdb/tdb_transactions.html
>>>
>>
> The limitations don't apply. The rest looks right.
>
> Needs cleaning up to remove historical TDB1 stuff but the txn code is much
> the same.
>
> Could you send me the complete thread dump as an attachment please?
> (Not knowing app the code it may be too hard to decipher and I don't know
> when I will get to look at it if it isn't quickly providing a clue)
>
>
> Andy
>
>
>>> With TDB2, is it bad to share a Dataset across threads and transactions ?
>>> What are the don't 's regarding TDB2 API use ?
>>>
>>>
>>> Andy
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> * at
>>>>>
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
>>>>> at
>>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>>> acquireWriterLock(TransactionCoordinator.java:305)
>>>>> at
>>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>>> begin(TransactionCoordinator.java:483)
>>>>> at
>>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>>> begin(TransactionCoordinator.java:450)
>>>>> at
>>>>> org.apache.jena.dboe.transaction.txn.TransactionalBase.begin
>>>>> (TransactionalBase.java:101)
>>>>> at
>>>>> org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGrap
>>>>> hTDB.java:418)
>>>>> at
>>>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>>>> tGraphWrapper.java:187)
>>>>> at
>>>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>>>> tGraphWrapper.java:187)
>>>>> at
>>>>> org.apache.jena.query.text.DatasetGraphText.begin(DatasetGra
>>>>> phText.java:115)
>>>>> at
>>>>> org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
>>>>> at
>>>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(Jena
>>>>> DatasetStore.scala:23)
>>>>> at scala.util.Try$.apply(Try.scala:192) at
>>>>> org.w3.banana.jena.JenaDatasetStore.rw
>>>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>>>> tore.scala:22)
>>>>> at org.w3.banana.jena.JenaDatasetStore.rw
>>>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>>>> tore.scala:10)*
>>>>>
>>>>> at
>>>>> deductions.runtime.services.ApplicationFacadeImpl$class.labe
>>>>> lForURITransaction(ApplicationFacadeImpl.scala:107)
>>>>> at
>>>>> controllers.Application$.labelForURITransaction(Application.scala:8)
>>>>> at
>>>>> controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
>>>>> at
>>>>> controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
>>>>> at
>>>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>>>> ply(WebPages.scala:64)
>>>>> at
>>>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>>>> ply(WebPages.scala:63)
>>>>> at
>>>>> play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>>>>
>>>>> It looks like a dead lock.
>>>>> The main database, concerned by these stacks, is not big: 181,799
>>>>> quads.
>>>>>
>>>>> uname -a
>>>>> Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
>>>>> x86_64 x86_64 GNU/Linux
>>>>>
>>>>> $ java -version
>>>>> openjdk version "1.8.0_151"
>>>>> OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.1
>>>>> 6.04.2-b12)
>>>>> OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
>>>>>
>>>>>
>>>>> I restarted the server, so that anyone can try.
>>>>>
>>>>> I forgot to say that the code changes are very limited:
>>>>>
>>>>> - switch for TDB2
>>>>> val dts =
>>>>> if (useTDB2)
>>>>>
>>>>> org.apache.jena.tdb2.TDB2Factory.connectDataset(database_
>>>>> location)
>>>>> else
>>>>>
>>>>> TDBFactory.createDataset(Paths.get(database_location).toString())
>>>>> - move an iterator inside a transaction after initial problem
>>>>> reported
>>>>>
>>>>> in first mail
>>>>>
>>>>>
>>>>> 2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>>>>
>>>>> Jean-Marc,
>>>>>
>>>>>>
>>>>>> Thank you for the report!
>>>>>>
>>>>>> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>>
>>>>>>> I started to test TBD2 on my application semantic_forms .
>>>>>>> I dumped the 3 databases with the TDB (1) tool,
>>>>>>> then loaded with the TDB 2 tool :
>>>>>>> runMain tdb2.tdbloader --loc TDB --verbose
>>>>>>> /home/jmv/src/semantic_forms/scala/dump.nq
>>>>>>>
>>>>>>> there where some bad quads with a literal subject from old bugs,
>>>>>>> that I
>>>>>>> had
>>>>>>> to manually edit out , but not a big deal.
>>>>>>>
>>>>>>> At runtime many things work, but I had one runtime error in
>>>>>>> /authenticate
>>>>>>> :
>>>>>>> *[TransactionException: Iterator used outside its original
>>>>>>> transaction]*
>>>>>>>
>>>>>>>
>>>>>>> yes - it's pickier than TDB1 :-)
>>>>>>
>>>>>>
>>>>>> Indeed , this is exactly that:
>>>>>>
>>>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>>>>>> forms/src/main/scala/deductions/runtime/services/Authenticat
>>>>>>> ion.scala#L66
>>>>>>>
>>>>>>>
>>>>>>> See Txn.calculateRead for executing a transaction and return a value.
>>>>>>
>>>>>>
>>>>>> This was on my laptop.
>>>>>>
>>>>>>> I'll fix the code, which seems to be changes compatible with old TDB.
>>>>>>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>>>>>>
>>>>>>> Hope I don't bother you with non complete report .
>>>>>>>
>>>>>>>
>>>>>>> Not at all.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Jean-Marc Vanel
>>> http://www.semantic-forms.cc:9111/display?displayuri=http:/
>>> /jmvanel.free.fr/jmv.rdf%23me#subject
>>> <http://www.semantic-forms.cc:9111/display?displayuri=http:/
>>> /jmvanel.free.fr/jmv.rdf%23me>
>>> Déductions SARL - Consulting, services, training,
>>> Rule-based programming, Semantic Web
>>> +33 (0)6 89 16 29 52 <+33%206%2089%2016%2029%2052>
>>> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>>>
>>>
>>
>>
>>
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Andy Seaborne <an...@apache.org>.
On 16/11/17 17:17, Jean-Marc Vanel wrote:
> Sorry that was not clear , rewriting.
Good - because was wondering who wrote what :-)
>
> 2017-11-16 17:36 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>
>>
>> 2017-11-16 16:11 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>
>>>
>>> On 16/11/17 09:12, Jean-Marc Vanel wrote:
>>>
>>>> So I tested on the sandbox site,
>>>> and it got quickly frozen :( .
>>>> ... <http://semantic-forms.cc:9111/>
>>>>
>>>> In shell server side, I did a kill -3 to get the threads dump,
>>>> and there are 8 occurences of the same pattern in bold:
>>>>
>>>> "application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
>>>> tid=0x00007fdb74007000 nid=0x5f5f waiting on condition
>>>> [0x00007fdb881b8000]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for <0x00000000f3f42378> (a
>>>>
>>>>
>>> java.util.concurrent.Semaphore$FairSync)
>>>
>>> Some other thread is holding the permit that control the number of
>>> writers.
>>>
>>> The WriterLock is a Semaphore with one permit.
>>>
>>> The question becomes who has the permit and are they making progress
>>>
>>> Note - the lock is not re-entrant. begin-begin on the same thread will
>>> deadlock (in case Akka is being clever about actors to threads).
>>>
>>
>> I assume from this that the error lies in my code, running fine with TDB1,
>> but TDB2 is mode picky .
>> I'm puzzled as to what to do.
>> Which bad practices to look for in the code ?
>> Instrumenting the code for run-time messages ?
>>
>> My understading is that Play! framework activates one thread from a pool
>> to process the HTTP request, so here everything is sequential.
>> I also start 2 or 3 futures, so each in another thread.
>>
> So concurrent lock acquire can occur between threads, but I understood that
> it's allowed, and even more efficient in TDB2 because several writes can
> occur concurrently.
TDB2 is like TDB1 single-active-writer.
> Embedded commits an single thread were always forbidden,
> and I don't think that there are some, but this is possible.
> When this happened in TDB1 , a TransactionException was thrown saying
> "allready in transaction".
> TDB2 does not throw in such case ?
I just checked it does.
So who is holding the permit and why aren't they giving it up?
Have they become blocked on something else ... which a waiting thread
has? (deadlock by starvation).
>
>> It's not clear what in TDB 1 doc is still good , and what not :
>> https://jena.apache.org/documentation/tdb/tdb_transactions.html
The limitations don't apply. The rest looks right.
Needs cleaning up to remove historical TDB1 stuff but the txn code is
much the same.
Could you send me the complete thread dump as an attachment please?
(Not knowing app the code it may be too hard to decipher and I don't
know when I will get to look at it if it isn't quickly providing a clue)
Andy
>>
>> With TDB2, is it bad to share a Dataset across threads and transactions ?
>> What are the don't 's regarding TDB2 API use ?
>>
>>
>>> Andy
>>>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> * at
>>>>
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at
>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>> acquireWriterLock(TransactionCoordinator.java:305)
>>>> at
>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>> begin(TransactionCoordinator.java:483)
>>>> at
>>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>>> begin(TransactionCoordinator.java:450)
>>>> at
>>>> org.apache.jena.dboe.transaction.txn.TransactionalBase.begin
>>>> (TransactionalBase.java:101)
>>>> at
>>>> org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGrap
>>>> hTDB.java:418)
>>>> at
>>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>>> tGraphWrapper.java:187)
>>>> at
>>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>>> tGraphWrapper.java:187)
>>>> at
>>>> org.apache.jena.query.text.DatasetGraphText.begin(DatasetGra
>>>> phText.java:115)
>>>> at
>>>> org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
>>>> at
>>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(Jena
>>>> DatasetStore.scala:23)
>>>> at scala.util.Try$.apply(Try.scala:192) at
>>>> org.w3.banana.jena.JenaDatasetStore.rw
>>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>>> tore.scala:22)
>>>> at org.w3.banana.jena.JenaDatasetStore.rw
>>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>>> tore.scala:10)*
>>>>
>>>> at
>>>> deductions.runtime.services.ApplicationFacadeImpl$class.labe
>>>> lForURITransaction(ApplicationFacadeImpl.scala:107)
>>>> at
>>>> controllers.Application$.labelForURITransaction(Application.scala:8)
>>>> at
>>>> controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
>>>> at
>>>> controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
>>>> at
>>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>>> ply(WebPages.scala:64)
>>>> at
>>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>>> ply(WebPages.scala:63)
>>>> at
>>>> play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>>>
>>>> It looks like a dead lock.
>>>> The main database, concerned by these stacks, is not big: 181,799 quads.
>>>>
>>>> uname -a
>>>> Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
>>>> x86_64 x86_64 GNU/Linux
>>>>
>>>> $ java -version
>>>> openjdk version "1.8.0_151"
>>>> OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.1
>>>> 6.04.2-b12)
>>>> OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
>>>>
>>>>
>>>> I restarted the server, so that anyone can try.
>>>>
>>>> I forgot to say that the code changes are very limited:
>>>>
>>>> - switch for TDB2
>>>> val dts =
>>>> if (useTDB2)
>>>>
>>>> org.apache.jena.tdb2.TDB2Factory.connectDataset(database_location)
>>>> else
>>>>
>>>> TDBFactory.createDataset(Paths.get(database_location).toString())
>>>> - move an iterator inside a transaction after initial problem
>>>> reported
>>>>
>>>> in first mail
>>>>
>>>>
>>>> 2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>>>
>>>> Jean-Marc,
>>>>>
>>>>> Thank you for the report!
>>>>>
>>>>> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>>>>>
>>>>> Hi
>>>>>>
>>>>>> I started to test TBD2 on my application semantic_forms .
>>>>>> I dumped the 3 databases with the TDB (1) tool,
>>>>>> then loaded with the TDB 2 tool :
>>>>>> runMain tdb2.tdbloader --loc TDB --verbose
>>>>>> /home/jmv/src/semantic_forms/scala/dump.nq
>>>>>>
>>>>>> there where some bad quads with a literal subject from old bugs, that I
>>>>>> had
>>>>>> to manually edit out , but not a big deal.
>>>>>>
>>>>>> At runtime many things work, but I had one runtime error in
>>>>>> /authenticate
>>>>>> :
>>>>>> *[TransactionException: Iterator used outside its original
>>>>>> transaction]*
>>>>>>
>>>>>>
>>>>> yes - it's pickier than TDB1 :-)
>>>>>
>>>>>
>>>>> Indeed , this is exactly that:
>>>>>
>>>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>>>>> forms/src/main/scala/deductions/runtime/services/Authenticat
>>>>>> ion.scala#L66
>>>>>>
>>>>>>
>>>>> See Txn.calculateRead for executing a transaction and return a value.
>>>>>
>>>>>
>>>>> This was on my laptop.
>>>>>> I'll fix the code, which seems to be changes compatible with old TDB.
>>>>>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>>>>>
>>>>>> Hope I don't bother you with non complete report .
>>>>>>
>>>>>>
>>>>> Not at all.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>
>>
>> --
>> Jean-Marc Vanel
>> http://www.semantic-forms.cc:9111/display?displayuri=http:/
>> /jmvanel.free.fr/jmv.rdf%23me#subject
>> <http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
>> Déductions SARL - Consulting, services, training,
>> Rule-based programming, Semantic Web
>> +33 (0)6 89 16 29 52 <+33%206%2089%2016%2029%2052>
>> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>>
>
>
>
Re: Testing TBD2: first report: code changes to be made
Posted by Jean-Marc Vanel <je...@gmail.com>.
Sorry that was not clear , rewriting.
2017-11-16 17:36 GMT+01:00 Jean-Marc Vanel <je...@gmail.com>:
>
> 2017-11-16 16:11 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>>
>> On 16/11/17 09:12, Jean-Marc Vanel wrote:
>>
>>> So I tested on the sandbox site,
>>> and it got quickly frozen :( .
>>> ... <http://semantic-forms.cc:9111/>
>>>
>>> In shell server side, I did a kill -3 to get the threads dump,
>>> and there are 8 occurences of the same pattern in bold:
>>>
>>> "application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
>>> tid=0x00007fdb74007000 nid=0x5f5f waiting on condition
>>> [0x00007fdb881b8000]
>>> java.lang.Thread.State: WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for <0x00000000f3f42378> (a
>>>
>>>
>> java.util.concurrent.Semaphore$FairSync)
>>
>> Some other thread is holding the permit that control the number of
>> writers.
>>
>> The WriterLock is a Semaphore with one permit.
>>
>> The question becomes who has the permit and are they making progress
>>
>> Note - the lock is not re-entrant. begin-begin on the same thread will
>> deadlock (in case Akka is being clever about actors to threads).
>>
>
> I assume from this that the error lies in my code, running fine with TDB1,
> but TDB2 is mode picky .
> I'm puzzled as to what to do.
> Which bad practices to look for in the code ?
> Instrumenting the code for run-time messages ?
>
> My understading is that Play! framework activates one thread from a pool
> to process the HTTP request, so here everything is sequential.
> I also start 2 or 3 futures, so each in another thread.
>
So concurrent lock acquire can occur between threads, but I understood that
it's allowed, and even more efficient in TDB2 because several writes can
occur concurrently.
Embedded commits an single thread were always forbidden,
and I don't think that there are some, but this is possible.
When this happened in TDB1 , a TransactionException was thrown saying
"allready in transaction".
TDB2 does not throw in such case ?
> It's not clear what in TDB 1 doc is still good , and what not :
> https://jena.apache.org/documentation/tdb/tdb_transactions.html
>
> With TDB2, is it bad to share a Dataset across threads and transactions ?
> What are the don't 's regarding TDB2 API use ?
>
>
>> Andy
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> * at
>>>
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at
>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>> acquireWriterLock(TransactionCoordinator.java:305)
>>> at
>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>> begin(TransactionCoordinator.java:483)
>>> at
>>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>>> begin(TransactionCoordinator.java:450)
>>> at
>>> org.apache.jena.dboe.transaction.txn.TransactionalBase.begin
>>> (TransactionalBase.java:101)
>>> at
>>> org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGrap
>>> hTDB.java:418)
>>> at
>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>> tGraphWrapper.java:187)
>>> at
>>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>>> tGraphWrapper.java:187)
>>> at
>>> org.apache.jena.query.text.DatasetGraphText.begin(DatasetGra
>>> phText.java:115)
>>> at
>>> org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
>>> at
>>> org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(Jena
>>> DatasetStore.scala:23)
>>> at scala.util.Try$.apply(Try.scala:192) at
>>> org.w3.banana.jena.JenaDatasetStore.rw
>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>> tore.scala:22)
>>> at org.w3.banana.jena.JenaDatasetStore.rw
>>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>>> tore.scala:10)*
>>>
>>> at
>>> deductions.runtime.services.ApplicationFacadeImpl$class.labe
>>> lForURITransaction(ApplicationFacadeImpl.scala:107)
>>> at
>>> controllers.Application$.labelForURITransaction(Application.scala:8)
>>> at
>>> controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
>>> at
>>> controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
>>> at
>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>> ply(WebPages.scala:64)
>>> at
>>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>>> ply(WebPages.scala:63)
>>> at
>>> play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>>
>>> It looks like a dead lock.
>>> The main database, concerned by these stacks, is not big: 181,799 quads.
>>>
>>> uname -a
>>> Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
>>> x86_64 x86_64 GNU/Linux
>>>
>>> $ java -version
>>> openjdk version "1.8.0_151"
>>> OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.1
>>> 6.04.2-b12)
>>> OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
>>>
>>>
>>> I restarted the server, so that anyone can try.
>>>
>>> I forgot to say that the code changes are very limited:
>>>
>>> - switch for TDB2
>>> val dts =
>>> if (useTDB2)
>>>
>>> org.apache.jena.tdb2.TDB2Factory.connectDataset(database_location)
>>> else
>>>
>>> TDBFactory.createDataset(Paths.get(database_location).toString())
>>> - move an iterator inside a transaction after initial problem
>>> reported
>>>
>>> in first mail
>>>
>>>
>>> 2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>>
>>> Jean-Marc,
>>>>
>>>> Thank you for the report!
>>>>
>>>> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>>>>
>>>> Hi
>>>>>
>>>>> I started to test TBD2 on my application semantic_forms .
>>>>> I dumped the 3 databases with the TDB (1) tool,
>>>>> then loaded with the TDB 2 tool :
>>>>> runMain tdb2.tdbloader --loc TDB --verbose
>>>>> /home/jmv/src/semantic_forms/scala/dump.nq
>>>>>
>>>>> there where some bad quads with a literal subject from old bugs, that I
>>>>> had
>>>>> to manually edit out , but not a big deal.
>>>>>
>>>>> At runtime many things work, but I had one runtime error in
>>>>> /authenticate
>>>>> :
>>>>> *[TransactionException: Iterator used outside its original
>>>>> transaction]*
>>>>>
>>>>>
>>>> yes - it's pickier than TDB1 :-)
>>>>
>>>>
>>>> Indeed , this is exactly that:
>>>>
>>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>>>> forms/src/main/scala/deductions/runtime/services/Authenticat
>>>>> ion.scala#L66
>>>>>
>>>>>
>>>> See Txn.calculateRead for executing a transaction and return a value.
>>>>
>>>>
>>>> This was on my laptop.
>>>>> I'll fix the code, which seems to be changes compatible with old TDB.
>>>>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>>>>
>>>>> Hope I don't bother you with non complete report .
>>>>>
>>>>>
>>>> Not at all.
>>>>
>>>> Andy
>>>>
>>>>
>>>
>>>
>>>
>
>
> --
> Jean-Marc Vanel
> http://www.semantic-forms.cc:9111/display?displayuri=http:/
> /jmvanel.free.fr/jmv.rdf%23me#subject
> <http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
> Déductions SARL - Consulting, services, training,
> Rule-based programming, Semantic Web
> +33 (0)6 89 16 29 52 <+33%206%2089%2016%2029%2052>
> Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
>
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Jean-Marc Vanel <je...@gmail.com>.
2017-11-16 16:11 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
> On 16/11/17 09:12, Jean-Marc Vanel wrote:
>
>> So I tested on the sandbox site,
>> and it got quickly frozen :( .
>> ... <http://semantic-forms.cc:9111/>
>>
>> In shell server side, I did a kill -3 to get the threads dump,
>> and there are 8 occurences of the same pattern in bold:
>>
>> "application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
>> tid=0x00007fdb74007000 nid=0x5f5f waiting on condition
>> [0x00007fdb881b8000]
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000000f3f42378> (a
>>
>>
> java.util.concurrent.Semaphore$FairSync)
>
> Some other thread is holding the permit that control the number of writers.
>
> The WriterLock is a Semaphore with one permit.
>
> The question becomes who has the permit and are they making progress
>
> Note - the lock is not re-entrant. begin-begin on the same thread will
> deadlock (in case Akka is being clever about actors to threads).
>
I assume from this that the error lies in my code, running fine with TDB1,
but TDB2 is mode picky .
I'm puzzled as to what to do.
Which bad practices to look for in the code ?
Instrumenting the code for run-time messages ?
My understading is that Play! framework activates one thread from a pool to
process the HTTP request, so here everything is sequential.
I also start 2 or 3 futures, so each in another thread.
I don't see where concurrent lock acquire can occur.
It's not clear what in TDB 1 doc is still good , and what not :
https://jena.apache.org/documentation/tdb/tdb_transactions.html
With TDB2, is it bad to share a Dataset across threads and transactions ?
What are the don't 's regarding TDB2 API use ?
> Andy
>
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> * at
>>
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>> dCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at
>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>> acquireWriterLock(TransactionCoordinator.java:305)
>> at
>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>> begin(TransactionCoordinator.java:483)
>> at
>> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.
>> begin(TransactionCoordinator.java:450)
>> at
>> org.apache.jena.dboe.transaction.txn.TransactionalBase.
>> begin(TransactionalBase.java:101)
>> at
>> org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGrap
>> hTDB.java:418)
>> at
>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>> tGraphWrapper.java:187)
>> at
>> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(Datase
>> tGraphWrapper.java:187)
>> at
>> org.apache.jena.query.text.DatasetGraphText.begin(DatasetGra
>> phText.java:115)
>> at
>> org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
>> at
>> org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(
>> JenaDatasetStore.scala:23)
>> at scala.util.Try$.apply(Try.scala:192) at
>> org.w3.banana.jena.JenaDatasetStore.rw
>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>> tore.scala:22)
>> at org.w3.banana.jena.JenaDatasetStore.rw
>> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetS
>> tore.scala:10)*
>>
>> at
>> deductions.runtime.services.ApplicationFacadeImpl$class.labe
>> lForURITransaction(ApplicationFacadeImpl.scala:107)
>> at
>> controllers.Application$.labelForURITransaction(Application.scala:8)
>> at
>> controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
>> at
>> controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
>> at
>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>> ply(WebPages.scala:64)
>> at
>> controllers.WebPages$$anonfun$outputMainPageWithContent$1.ap
>> ply(WebPages.scala:63)
>> at
>> play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>>
>> It looks like a dead lock.
>> The main database, concerned by these stacks, is not big: 181,799 quads.
>>
>> uname -a
>> Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
>> x86_64 x86_64 GNU/Linux
>>
>> $ java -version
>> openjdk version "1.8.0_151"
>> OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.1
>> 6.04.2-b12)
>> OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
>>
>>
>> I restarted the server, so that anyone can try.
>>
>> I forgot to say that the code changes are very limited:
>>
>> - switch for TDB2
>> val dts =
>> if (useTDB2)
>>
>> org.apache.jena.tdb2.TDB2Factory.connectDataset(database_location)
>> else
>>
>> TDBFactory.createDataset(Paths.get(database_location).toString())
>> - move an iterator inside a transaction after initial problem reported
>>
>> in first mail
>>
>>
>> 2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
>>
>> Jean-Marc,
>>>
>>> Thank you for the report!
>>>
>>> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>>>
>>> Hi
>>>>
>>>> I started to test TBD2 on my application semantic_forms .
>>>> I dumped the 3 databases with the TDB (1) tool,
>>>> then loaded with the TDB 2 tool :
>>>> runMain tdb2.tdbloader --loc TDB --verbose
>>>> /home/jmv/src/semantic_forms/scala/dump.nq
>>>>
>>>> there where some bad quads with a literal subject from old bugs, that I
>>>> had
>>>> to manually edit out , but not a big deal.
>>>>
>>>> At runtime many things work, but I had one runtime error in
>>>> /authenticate
>>>> :
>>>> *[TransactionException: Iterator used outside its original transaction]*
>>>>
>>>>
>>> yes - it's pickier than TDB1 :-)
>>>
>>>
>>> Indeed , this is exactly that:
>>>
>>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>>> forms/src/main/scala/deductions/runtime/services/Authenticat
>>>> ion.scala#L66
>>>>
>>>>
>>> See Txn.calculateRead for executing a transaction and return a value.
>>>
>>>
>>> This was on my laptop.
>>>> I'll fix the code, which seems to be changes compatible with old TDB.
>>>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>>>
>>>> Hope I don't bother you with non complete report .
>>>>
>>>>
>>> Not at all.
>>>
>>> Andy
>>>
>>>
>>
>>
>>
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Andy Seaborne <an...@apache.org>.
On 16/11/17 09:12, Jean-Marc Vanel wrote:
> So I tested on the sandbox site,
> and it got quickly frozen :( .
>
> In 3 tabs of the browser,
> I sent requests more or less simultaneously:
> http://semantic-forms.cc:9111/search?q=chrysophylla&clas=
> http://semantic-forms.cc:9111/display?displayuri=http%3A%2F%2Fsemantic-forms.cc%3A9111%2Fldp%2F1510064559306-25745506095233869
> http://semantic-forms.cc:9111/
>
> In shell server side, I did a kill -3 to get the threads dump,
> and there are 8 occurences of the same pattern in bold:
>
> "application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
> tid=0x00007fdb74007000 nid=0x5f5f waiting on condition [0x00007fdb881b8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000f3f42378> (a
>
java.util.concurrent.Semaphore$FairSync)
Some other thread is holding the permit that control the number of writers.
The WriterLock is a Semaphore with one permit.
The question becomes who has the permit and are they making progress
Note - the lock is not re-entrant. begin-begin on the same thread will
deadlock (in case Akka is being clever about actors to threads).
Andy
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> * at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at
> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.acquireWriterLock(TransactionCoordinator.java:305)
> at
> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.begin(TransactionCoordinator.java:483)
> at
> org.apache.jena.dboe.transaction.txn.TransactionCoordinator.begin(TransactionCoordinator.java:450)
> at
> org.apache.jena.dboe.transaction.txn.TransactionalBase.begin(TransactionalBase.java:101)
> at
> org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGraphTDB.java:418)
> at
> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(DatasetGraphWrapper.java:187)
> at
> org.apache.jena.sparql.core.DatasetGraphWrapper.begin(DatasetGraphWrapper.java:187)
> at
> org.apache.jena.query.text.DatasetGraphText.begin(DatasetGraphText.java:115)
> at
> org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
> at
> org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(JenaDatasetStore.scala:23)
> at scala.util.Try$.apply(Try.scala:192) at
> org.w3.banana.jena.JenaDatasetStore.rw
> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetStore.scala:22)
> at org.w3.banana.jena.JenaDatasetStore.rw
> <http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetStore.scala:10)*
> at
> deductions.runtime.services.ApplicationFacadeImpl$class.labelForURITransaction(ApplicationFacadeImpl.scala:107)
> at
> controllers.Application$.labelForURITransaction(Application.scala:8)
> at
> controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
> at
> controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
> at
> controllers.WebPages$$anonfun$outputMainPageWithContent$1.apply(WebPages.scala:64)
> at
> controllers.WebPages$$anonfun$outputMainPageWithContent$1.apply(WebPages.scala:63)
> at
> play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
>
> It looks like a dead lock.
> The main database, concerned by these stacks, is not big: 181,799 quads.
>
> uname -a
> Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
> x86_64 x86_64 GNU/Linux
>
> $ java -version
> openjdk version "1.8.0_151"
> OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
> OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
>
>
> I restarted the server, so that anyone can try.
>
> I forgot to say that the code changes are very limited:
>
> - switch for TDB2
> val dts =
> if (useTDB2)
>
> org.apache.jena.tdb2.TDB2Factory.connectDataset(database_location)
> else
>
> TDBFactory.createDataset(Paths.get(database_location).toString())
> - move an iterator inside a transaction after initial problem reported
> in first mail
>
>
> 2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
>
>> Jean-Marc,
>>
>> Thank you for the report!
>>
>> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>>
>>> Hi
>>>
>>> I started to test TBD2 on my application semantic_forms .
>>> I dumped the 3 databases with the TDB (1) tool,
>>> then loaded with the TDB 2 tool :
>>> runMain tdb2.tdbloader --loc TDB --verbose
>>> /home/jmv/src/semantic_forms/scala/dump.nq
>>>
>>> there where some bad quads with a literal subject from old bugs, that I
>>> had
>>> to manually edit out , but not a big deal.
>>>
>>> At runtime many things work, but I had one runtime error in /authenticate
>>> :
>>> *[TransactionException: Iterator used outside its original transaction]*
>>>
>>
>> yes - it's pickier than TDB1 :-)
>>
>>
>> Indeed , this is exactly that:
>>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>>> forms/src/main/scala/deductions/runtime/services/Authentication.scala#L66
>>>
>>
>> See Txn.calculateRead for executing a transaction and return a value.
>>
>>
>>> This was on my laptop.
>>> I'll fix the code, which seems to be changes compatible with old TDB.
>>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>>
>>> Hope I don't bother you with non complete report .
>>>
>>
>> Not at all.
>>
>> Andy
>>
>
>
>
Re: Testing TBD2: first report: code changes to be made
Posted by Jean-Marc Vanel <je...@gmail.com>.
So I tested on the sandbox site,
and it got quickly frozen :( .
In 3 tabs of the browser,
I sent requests more or less simultaneously:
http://semantic-forms.cc:9111/search?q=chrysophylla&clas=
http://semantic-forms.cc:9111/display?displayuri=http%3A%2F%2Fsemantic-forms.cc%3A9111%2Fldp%2F1510064559306-25745506095233869
http://semantic-forms.cc:9111/
In shell server side, I did a kill -3 to get the threads dump,
and there are 8 occurences of the same pattern in bold:
"application-akka.actor.default-dispatcher-13" #38 prio=5 os_prio=0
tid=0x00007fdb74007000 nid=0x5f5f waiting on condition [0x00007fdb881b8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f3f42378> (a
java.util.concurrent.Semaphore$FairSync)
* at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at
org.apache.jena.dboe.transaction.txn.TransactionCoordinator.acquireWriterLock(TransactionCoordinator.java:305)
at
org.apache.jena.dboe.transaction.txn.TransactionCoordinator.begin(TransactionCoordinator.java:483)
at
org.apache.jena.dboe.transaction.txn.TransactionCoordinator.begin(TransactionCoordinator.java:450)
at
org.apache.jena.dboe.transaction.txn.TransactionalBase.begin(TransactionalBase.java:101)
at
org.apache.jena.tdb2.store.DatasetGraphTDB.begin(DatasetGraphTDB.java:418)
at
org.apache.jena.sparql.core.DatasetGraphWrapper.begin(DatasetGraphWrapper.java:187)
at
org.apache.jena.sparql.core.DatasetGraphWrapper.begin(DatasetGraphWrapper.java:187)
at
org.apache.jena.query.text.DatasetGraphText.begin(DatasetGraphText.java:115)
at
org.apache.jena.sparql.core.DatasetImpl.begin(DatasetImpl.java:111)
at
org.w3.banana.jena.JenaDatasetStore$$anonfun$rw$1.apply(JenaDatasetStore.scala:23)
at scala.util.Try$.apply(Try.scala:192) at
org.w3.banana.jena.JenaDatasetStore.rw
<http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetStore.scala:22)
at org.w3.banana.jena.JenaDatasetStore.rw
<http://org.w3.banana.jena.JenaDatasetStore.rw>(JenaDatasetStore.scala:10)*
at
deductions.runtime.services.ApplicationFacadeImpl$class.labelForURITransaction(ApplicationFacadeImpl.scala:107)
at
controllers.Application$.labelForURITransaction(Application.scala:8)
at
controllers.WebPages$MainPagePrecomputePlay.<init>(WebPages.scala:46)
at
controllers.WebPages$MainPagePrecomputePlay$.apply(WebPages.scala:39)
at
controllers.WebPages$$anonfun$outputMainPageWithContent$1.apply(WebPages.scala:64)
at
controllers.WebPages$$anonfun$outputMainPageWithContent$1.apply(WebPages.scala:63)
at
play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
It looks like a dead lock.
The main database, concerned by these stacks, is not big: 181,799 quads.
uname -a
Linux scw-80de9e 4.5.7-std-3 #1 SMP Tue Jul 12 09:56:30 UTC 2016 x86_64
x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
I restarted the server, so that anyone can try.
I forgot to say that the code changes are very limited:
- switch for TDB2
val dts =
if (useTDB2)
org.apache.jena.tdb2.TDB2Factory.connectDataset(database_location)
else
TDBFactory.createDataset(Paths.get(database_location).toString())
- move an iterator inside a transaction after initial problem reported
in first mail
2017-11-15 17:46 GMT+01:00 Andy Seaborne <an...@apache.org>:
> Jean-Marc,
>
> Thank you for the report!
>
> On 15/11/17 11:18, Jean-Marc Vanel wrote:
>
>> Hi
>>
>> I started to test TBD2 on my application semantic_forms .
>> I dumped the 3 databases with the TDB (1) tool,
>> then loaded with the TDB 2 tool :
>> runMain tdb2.tdbloader --loc TDB --verbose
>> /home/jmv/src/semantic_forms/scala/dump.nq
>>
>> there where some bad quads with a literal subject from old bugs, that I
>> had
>> to manually edit out , but not a big deal.
>>
>> At runtime many things work, but I had one runtime error in /authenticate
>> :
>> *[TransactionException: Iterator used outside its original transaction]*
>>
>
> yes - it's pickier than TDB1 :-)
>
>
> Indeed , this is exactly that:
>> https://github.com/jmvanel/semantic_forms/blob/master/scala/
>> forms/src/main/scala/deductions/runtime/services/Authentication.scala#L66
>>
>
> See Txn.calculateRead for executing a transaction and return a value.
>
>
>> This was on my laptop.
>> I'll fix the code, which seems to be changes compatible with old TDB.
>> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>>
>> Hope I don't bother you with non complete report .
>>
>
> Not at all.
>
> Andy
>
--
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject
<http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me>
Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui
Re: Testing TBD2: first report: code changes to be made
Posted by Andy Seaborne <an...@apache.org>.
Jean-Marc,
Thank you for the report!
On 15/11/17 11:18, Jean-Marc Vanel wrote:
> Hi
>
> I started to test TBD2 on my application semantic_forms .
> I dumped the 3 databases with the TDB (1) tool,
> then loaded with the TDB 2 tool :
> runMain tdb2.tdbloader --loc TDB --verbose
> /home/jmv/src/semantic_forms/scala/dump.nq
>
> there where some bad quads with a literal subject from old bugs, that I had
> to manually edit out , but not a big deal.
>
> At runtime many things work, but I had one runtime error in /authenticate :
> *[TransactionException: Iterator used outside its original transaction]*
yes - it's pickier than TDB1 :-)
> Indeed , this is exactly that:
> https://github.com/jmvanel/semantic_forms/blob/master/scala/forms/src/main/scala/deductions/runtime/services/Authentication.scala#L66
See Txn.calculateRead for executing a transaction and return a value.
>
> This was on my laptop.
> I'll fix the code, which seems to be changes compatible with old TDB.
> Then I'll upgrade to TDB2 on the sandbox, then on the real site.
>
> Hope I don't bother you with non complete report .
Not at all.
Andy