You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@clerezza.apache.org by Minto van der Sluis <mi...@xup.nl> on 2014/01/14 09:22:12 UTC
ConcurrentModificationException
Hi Folks,
A reply [1] from Andy to one of my mails triggered me. I looked up the
"serializable transactions" that Andy mentioned [2]. Especially the
"Multi-threaded use" section caught my attention. If I read it correctly
using a single dataset in multiple threads is discouraged.
Looking at Clerezza's use of Jena's dataset I see no sign of
multi-threading support. Might this be the cause of our
ConcurrentModificationException perils? Am I on to something here?
Please share your thoughts.
Regards,
Minto
[1]
http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
[2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
Re: ConcurrentModificationException
Posted by Minto van der Sluis <mi...@xup.nl>.
Hi Andy,
Thanks for your clarification
Andy Seaborne schreef op 15-1-2014 0:37:
> (seem to have forgotten to "send" on an earlier version)
>
> On 14/01/14 10:59, Minto van der Sluis wrote:
>> @Andy
>>
>> You're remark puzzles me. I know it's a quote from [1]. But is it to
>> emphasize that I am correct or to tell me I'm incorrent?
>
> To my reading, your text is blurring the dataset/transaction distinction.
>
> e.g. Fuseki has one dataset and provide multiple transactions (each
> request is a transaction) all the time.
>
>>
>> dataset != transaction, but the example also shows that every thread has
>> its own dataset.
>
> It's supposed to show getting a dataset somehow - not necessarily
> creat it.
>
> In fact, there is only one internal object per location (per dataset
> on disk). That's how coordination of multiple threads is done.
> TDBFactory may returns different wrapper objects but inside there is
> one master object per location.
>
> Andy
>
>>
>> Regards,
>>
>> Minto
>>
>> Andy Seaborne schreef op 14-1-2014 10:45:
>>> On 14/01/14 08:22, Minto van der Sluis wrote:
>>>> Hi Folks,
>>>>
>>>> A reply [1] from Andy to one of my mails triggered me. I looked up
>>>> the
>>>> "serializable transactions" that Andy mentioned [2]. Especially the
>>>> "Multi-threaded use" section caught my attention. If I read it
>>>> correctly
>>>> using a single dataset in multiple threads is discouraged.
>>>
>>> """
>>> While it is possible to share a *transaction* between multiple
>>> threads, this is not encouraged.
>>> """
>>>
>>>>
>>>> Looking at Clerezza's use of Jena's dataset I see no sign of
>>>> multi-threading support. Might this be the cause of our
>>>> ConcurrentModificationException perils? Am I on to something here?
>>>>
>>>> Please share your thoughts.
>>>>
>>>> Regards,
>>>>
>>>> Minto
>>>>
>>>> [1]
>>>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
>>>>
>>>>
>>>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>
--
ir. ing. Minto van der Sluis
Software innovator / renovator
Xup BV
Mobiel: +31 (0) 626 014541
Re: ConcurrentModificationException
Posted by Andy Seaborne <an...@apache.org>.
(seem to have forgotten to "send" on an earlier version)
On 14/01/14 10:59, Minto van der Sluis wrote:
> @Andy
>
> You're remark puzzles me. I know it's a quote from [1]. But is it to
> emphasize that I am correct or to tell me I'm incorrent?
To my reading, your text is blurring the dataset/transaction distinction.
e.g. Fuseki has one dataset and provide multiple transactions (each
request is a transaction) all the time.
>
> dataset != transaction, but the example also shows that every thread has
> its own dataset.
It's supposed to show getting a dataset somehow - not necessarily creat it.
In fact, there is only one internal object per location (per dataset on
disk). That's how coordination of multiple threads is done. TDBFactory
may returns different wrapper objects but inside there is one master
object per location.
Andy
>
> Regards,
>
> Minto
>
> Andy Seaborne schreef op 14-1-2014 10:45:
>> On 14/01/14 08:22, Minto van der Sluis wrote:
>>> Hi Folks,
>>>
>>> A reply [1] from Andy to one of my mails triggered me. I looked up the
>>> "serializable transactions" that Andy mentioned [2]. Especially the
>>> "Multi-threaded use" section caught my attention. If I read it correctly
>>> using a single dataset in multiple threads is discouraged.
>>
>> """
>> While it is possible to share a *transaction* between multiple
>> threads, this is not encouraged.
>> """
>>
>>>
>>> Looking at Clerezza's use of Jena's dataset I see no sign of
>>> multi-threading support. Might this be the cause of our
>>> ConcurrentModificationException perils? Am I on to something here?
>>>
>>> Please share your thoughts.
>>>
>>> Regards,
>>>
>>> Minto
>>>
>>> [1]
>>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
>>>
>>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>>
>>>
>>
>>
>>
>
>
Re: ConcurrentModificationException
Posted by Minto van der Sluis <mi...@xup.nl>.
@Andy
You're remark puzzles me. I know it's a quote from [1]. But is it to
emphasize that I am correct or to tell me I'm incorrent?
dataset != transaction, but the example also shows that every thread has
its own dataset.
Regards,
Minto
Andy Seaborne schreef op 14-1-2014 10:45:
> On 14/01/14 08:22, Minto van der Sluis wrote:
>> Hi Folks,
>>
>> A reply [1] from Andy to one of my mails triggered me. I looked up the
>> "serializable transactions" that Andy mentioned [2]. Especially the
>> "Multi-threaded use" section caught my attention. If I read it correctly
>> using a single dataset in multiple threads is discouraged.
>
> """
> While it is possible to share a *transaction* between multiple
> threads, this is not encouraged.
> """
>
>>
>> Looking at Clerezza's use of Jena's dataset I see no sign of
>> multi-threading support. Might this be the cause of our
>> ConcurrentModificationException perils? Am I on to something here?
>>
>> Please share your thoughts.
>>
>> Regards,
>>
>> Minto
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
>>
>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>
>>
>
>
>
--
ir. ing. Minto van der Sluis
Software innovator / renovator
Xup BV
Mobiel: +31 (0) 626 014541
Re: ConcurrentModificationException
Posted by Andy Seaborne <an...@apache.org>.
On 14/01/14 08:22, Minto van der Sluis wrote:
> Hi Folks,
>
> A reply [1] from Andy to one of my mails triggered me. I looked up the
> "serializable transactions" that Andy mentioned [2]. Especially the
> "Multi-threaded use" section caught my attention. If I read it correctly
> using a single dataset in multiple threads is discouraged.
"""
While it is possible to share a *transaction* between multiple threads,
this is not encouraged.
"""
>
> Looking at Clerezza's use of Jena's dataset I see no sign of
> multi-threading support. Might this be the cause of our
> ConcurrentModificationException perils? Am I on to something here?
>
> Please share your thoughts.
>
> Regards,
>
> Minto
>
> [1]
> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>
>
Re: ConcurrentModificationException
Posted by Reto Gmür <re...@wymiwyg.com>.
On Tue, Jan 14, 2014 at 2:53 PM, Andy Seaborne <an...@apache.org> wrote:
> On 14/01/14 09:22, Reto Gmür wrote:
>
>> Well, TDB should work with transaction but also with no concurrent access.
>> The latter is what clerezza supports and enforces. It supports it in that
>> while iterating one should aquire a read lock. It enforces in that every
>> write operation automatically aquires a write-lock on the dataset.
>>
>
> Test case please!
>
> There are test testing multithreaded access. Not sure what test you mean
and what about CLEREZZA-691?
>
The open issue is not a lack of locking but a deadlock with addAll (when
invoked with another graph from the same dataset as target). The solution,
analogous to what is done in jena, is to copy the added graph to memory and
then add this collection of triples to the graph.
Cheers,
Reto
>
> Andy
>
>
>
>> Cheers,
>> Reto
>>
>>
>> On Tue, Jan 14, 2014 at 9:22 AM, Minto van der Sluis <mi...@xup.nl>
>> wrote:
>>
>> Hi Folks,
>>>
>>> A reply [1] from Andy to one of my mails triggered me. I looked up the
>>> "serializable transactions" that Andy mentioned [2]. Especially the
>>> "Multi-threaded use" section caught my attention. If I read it correctly
>>> using a single dataset in multiple threads is discouraged.
>>>
>>> Looking at Clerezza's use of Jena's dataset I see no sign of
>>> multi-threading support. Might this be the cause of our
>>> ConcurrentModificationException perils? Am I on to something here?
>>>
>>> Please share your thoughts.
>>>
>>> Regards,
>>>
>>> Minto
>>>
>>> [1]
>>>
>>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/
>>> 201401.mbox/%3C52D41088.5070802%40apache.org%3E
>>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>>
>>>
>>>
>>>
>>
>
Re: ConcurrentModificationException
Posted by Andy Seaborne <an...@apache.org>.
On 14/01/14 09:22, Reto Gmür wrote:
> Well, TDB should work with transaction but also with no concurrent access.
> The latter is what clerezza supports and enforces. It supports it in that
> while iterating one should aquire a read lock. It enforces in that every
> write operation automatically aquires a write-lock on the dataset.
Test case please!
and what about CLEREZZA-691?
Andy
>
> Cheers,
> Reto
>
>
> On Tue, Jan 14, 2014 at 9:22 AM, Minto van der Sluis <mi...@xup.nl> wrote:
>
>> Hi Folks,
>>
>> A reply [1] from Andy to one of my mails triggered me. I looked up the
>> "serializable transactions" that Andy mentioned [2]. Especially the
>> "Multi-threaded use" section caught my attention. If I read it correctly
>> using a single dataset in multiple threads is discouraged.
>>
>> Looking at Clerezza's use of Jena's dataset I see no sign of
>> multi-threading support. Might this be the cause of our
>> ConcurrentModificationException perils? Am I on to something here?
>>
>> Please share your thoughts.
>>
>> Regards,
>>
>> Minto
>>
>> [1]
>>
>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>
>>
>>
>
Re: ConcurrentModificationException (CME)
Posted by Reto Gmür <re...@wymiwyg.com>.
On Tue, Jan 14, 2014 at 12:20 PM, Minto van der Sluis <mi...@xup.nl> wrote:
> Since we experience CMEs either "no concurrent access" is broken or like
> Andy mentioned in [1] java iterators are abused.
>
> Even though it should not be necessary, according to "no concurrent
> access", my application still synchronized all access to TcManager. This
> additional synchonization did result in a significant drop in the number
> of CME occurrences. But I still have a few.
>
> Most occurrences happen while reading (executing a query). Should we
> also explicitly start read transactions? Currently
> BaseTdbTcProvider.executeSparqlQuery() does not include starting a read
> transaction.
>
> dataset.begin(ReadWrite.READ)
>
Did you see the my recent resolution of CLEREZZA-252? Not sure if using a
transaction in one place would help as long as the other access happens
without transactions.
Cheers,
Reto
>
>
> IMO this is either caused by abusing java iterators or the internal
> SyncThread. But on the other hand SyncThread also uses write locks.
>
> Am I the only one experiencing CMEs? Are there other application running
> with multiple concurrent users that make use of Clerezza TDB provider?
>
> Regards,
>
> Minto
>
>
> Reto Gmür schreef op 14-1-2014 10:22:
> > Well, TDB should work with transaction but also with no concurrent
> access.
> > The latter is what clerezza supports and enforces. It supports it in that
> > while iterating one should aquire a read lock. It enforces in that every
> > write operation automatically aquires a write-lock on the dataset.
> >
> > Cheers,
> > Reto
> >
> >
> > On Tue, Jan 14, 2014 at 9:22 AM, Minto van der Sluis <mi...@xup.nl>
> wrote:
> >
> >> Hi Folks,
> >>
> >> A reply [1] from Andy to one of my mails triggered me. I looked up the
> >> "serializable transactions" that Andy mentioned [2]. Especially the
> >> "Multi-threaded use" section caught my attention. If I read it correctly
> >> using a single dataset in multiple threads is discouraged.
> >>
> >> Looking at Clerezza's use of Jena's dataset I see no sign of
> >> multi-threading support. Might this be the cause of our
> >> ConcurrentModificationException perils? Am I on to something here?
> >>
> >> Please share your thoughts.
> >>
> >> Regards,
> >>
> >> Minto
> >>
> >> [1]
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
> >> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
> >>
> >>
> >>
>
>
> --
> ir. ing. Minto van der Sluis
> Software innovator / renovator
> Xup BV
>
> Mobiel: +31 (0) 626 014541
>
>
Re: ConcurrentModificationException (CME)
Posted by Minto van der Sluis <mi...@xup.nl>.
Since we experience CMEs either "no concurrent access" is broken or like
Andy mentioned in [1] java iterators are abused.
Even though it should not be necessary, according to "no concurrent
access", my application still synchronized all access to TcManager. This
additional synchonization did result in a significant drop in the number
of CME occurrences. But I still have a few.
Most occurrences happen while reading (executing a query). Should we
also explicitly start read transactions? Currently
BaseTdbTcProvider.executeSparqlQuery() does not include starting a read
transaction.
dataset.begin(ReadWrite.READ)
IMO this is either caused by abusing java iterators or the internal
SyncThread. But on the other hand SyncThread also uses write locks.
Am I the only one experiencing CMEs? Are there other application running
with multiple concurrent users that make use of Clerezza TDB provider?
Regards,
Minto
Reto Gmür schreef op 14-1-2014 10:22:
> Well, TDB should work with transaction but also with no concurrent access.
> The latter is what clerezza supports and enforces. It supports it in that
> while iterating one should aquire a read lock. It enforces in that every
> write operation automatically aquires a write-lock on the dataset.
>
> Cheers,
> Reto
>
>
> On Tue, Jan 14, 2014 at 9:22 AM, Minto van der Sluis <mi...@xup.nl> wrote:
>
>> Hi Folks,
>>
>> A reply [1] from Andy to one of my mails triggered me. I looked up the
>> "serializable transactions" that Andy mentioned [2]. Especially the
>> "Multi-threaded use" section caught my attention. If I read it correctly
>> using a single dataset in multiple threads is discouraged.
>>
>> Looking at Clerezza's use of Jena's dataset I see no sign of
>> multi-threading support. Might this be the cause of our
>> ConcurrentModificationException perils? Am I on to something here?
>>
>> Please share your thoughts.
>>
>> Regards,
>>
>> Minto
>>
>> [1]
>>
>> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
>> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>>
>>
>>
--
ir. ing. Minto van der Sluis
Software innovator / renovator
Xup BV
Mobiel: +31 (0) 626 014541
Re: ConcurrentModificationException
Posted by Reto Gmür <re...@wymiwyg.com>.
Well, TDB should work with transaction but also with no concurrent access.
The latter is what clerezza supports and enforces. It supports it in that
while iterating one should aquire a read lock. It enforces in that every
write operation automatically aquires a write-lock on the dataset.
Cheers,
Reto
On Tue, Jan 14, 2014 at 9:22 AM, Minto van der Sluis <mi...@xup.nl> wrote:
> Hi Folks,
>
> A reply [1] from Andy to one of my mails triggered me. I looked up the
> "serializable transactions" that Andy mentioned [2]. Especially the
> "Multi-threaded use" section caught my attention. If I read it correctly
> using a single dataset in multiple threads is discouraged.
>
> Looking at Clerezza's use of Jena's dataset I see no sign of
> multi-threading support. Might this be the cause of our
> ConcurrentModificationException perils? Am I on to something here?
>
> Please share your thoughts.
>
> Regards,
>
> Minto
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/clerezza-dev/201401.mbox/%3C52D41088.5070802%40apache.org%3E
> [2] http://jena.apache.org/documentation/tdb/tdb_transactions.html
>
>
>