You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "A. Soroka" <aj...@virginia.edu> on 2015/11/30 19:19:40 UTC
Model locks and Dataset locks
I noticed the other day that the ModelCom locking behavior seems completely independent of the locking behavior of the dataset that may underlie a Model. I’m a little worried about that with the advent of DatasetGraphInMemory, because I think the “ergonomics” could be confusing. I have a test case up here:
https://gist.github.com/ajs6f/3e87cd6f78ec3b4e27a1
The problem is that when used without transactions, DatasetGraphInMemory tries to “do what the user meant” by auto-wrapping ::add or ::delete operations in transactions. But this leads to a case where a thread can get a model from a DatasetGraphInMemory-backed dataset, acquire the write lock for that model with no trouble, but then block trying to add or delete anything from it because the mutation is blocked by some other thread elsewhere that has the dataset lock. What would seem less surprising to me would be for the first thread to block waiting to get the model write lock.
Does this seem too obscure? Is it worth trying to think about how ModelCom could respect the locking of an underlying Dataset, if one exists?
---
A. Soroka
The University of Virginia Library
Re: Model locks and Dataset locks
Posted by "A. Soroka" <aj...@virginia.edu>.
Hm, okay. But looking at `GraphView`, it is understandably not exposing any means to get at the `DatasetGraph` that supports it. I assume that we do _not_ want to change that. Would we install transaction control methods in `GraphView`, so that `ModelCom` could use them to respect transactions in an underlying `DatasetGraph`?
---
A. Soroka
The University of Virginia Library
> On Dec 2, 2015, at 10:24 AM, Andy Seaborne <an...@apache.org> wrote:
>
> On 02/12/15 13:40, A. Soroka wrote:
>>> Thought:
>>> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
>>
>> This is pretty much where I was going with my question. But I think it would require something going on in ModelCom that could distinguish between those instances from a dataset and those that are “free-standing”. I’m not sure how that happens— perhaps Model::supportsTransactions?
>
> yes - eventually models via their graph need to know about datasets. We could create some kind of "dataset of one" (actually there already is) to unify the architecture.
>
> A corner case is that a model can be in a general dataset but come from another one so it's in two+ at once. At the moment that's only possible for the general dataset; otherwise Dataset(Graph) == storage
>
> Avoid model transactions though:
>
> * It's "begin()" which needs transaction promotion (and a risk of system aborts) and that isn't current supported anywhere.
>
> * It confuses the message that real/persistent storage is datasets these days.
>
> * May suggest nested transactions. Dataset -> model.
>
> * SPARQL may be across models and or create them
>
>>
>>> The code took a read lock - not that this matters because it is not the dataset lock -
>>
>> No, the code is model.enterCriticalSection(false), a write lock. That’s why this is an interesting case. Well, interesting to me, anyway. {grin} That call is exactly the one I would want to block, but it doesn’t, because ModelCom knows nothing of the underlying dataset.
>>
>
> My bad. API before enums were around!
>
>
>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>>> On Dec 1, 2015, at 5:48 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>> JENA-957
>>> And this is related to the discussion on JENA-624 currently.
>>>
>>> DatasetGraphInMemory is a special case where it's lock (which is MR+SW not MRSW) is used internally. It is not the model lock is part of the issue here.
>>>
>>> We can encourage the use of transactions.
>>>
>>> It may be possible to make the lock interact with transactions:
>>>
>>> Thought:
>>> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
>>>
>>>
>>> The gist example mixes transactional and non-tranactional use.
>>>
>>> At L56 it's outside the write transaction so autocommit applies. As the W transaction never commits, the implicit second W transaction hangs.
>>>
>>> The code took a read lock - not that this matters because it is not the dataset lock -
>>>
>>> With locks, if it's a read lock, then write access is not checked - it's up to the application to behave.
>>>
>>> http://jena.apache.org/documentation/notes/concurrency-howto.html
>>>
>>> If it were the dataset lock, then in this case "things go wrong" because the app is manipulating the lock the transaction mechanism is using. I hope that if L54 was ds.getLock().enterCriticalSection(write) then it would block. A read lock from the dataset will let the thread through as it is MR+SW and then the Graph:add complain (??)
>>>
>>>
>>> Elsewhere, in TDB, with great care, it is possible to MRSW inside a transaction. The upper level APIs add the ThreadLocal control to keep the hidden object but there really is an explicit Transaction object (org.apache.jena.tdb.transaction.Transaction).
>>>
>>> Andy
>>>
>>>
>>> On 30/11/15 18:19, A. Soroka wrote:
>>>> I noticed the other day that the ModelCom locking behavior seems completely independent of the locking behavior of the dataset that may underlie a Model. I’m a little worried about that with the advent of DatasetGraphInMemory, because I think the “ergonomics” could be confusing. I have a test case up here:
>>>>
>>>> https://gist.github.com/ajs6f/3e87cd6f78ec3b4e27a1
>>>>
>>>> The problem is that when used without transactions, DatasetGraphInMemory tries to “do what the user meant” by auto-wrapping ::add or ::delete operations in transactions. But this leads to a case where a thread can get a model from a DatasetGraphInMemory-backed dataset, acquire the write lock for that model with no trouble, but then block trying to add or delete anything from it because the mutation is blocked by some other thread elsewhere that has the dataset lock. What would seem less surprising to me would be for the first thread to block waiting to get the model write lock.
>>>>
>>>> Does this seem too obscure? Is it worth trying to think about how ModelCom could respect the locking of an underlying Dataset, if one exists?
>>>>
>>>> ---
>>>> A. Soroka
>>>> The University of Virginia Library
>>>>
>>>
>>
>
Re: Model locks and Dataset locks
Posted by Andy Seaborne <an...@apache.org>.
On 02/12/15 13:40, A. Soroka wrote:
>> Thought:
>> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
>
> This is pretty much where I was going with my question. But I think it would require something going on in ModelCom that could distinguish between those instances from a dataset and those that are “free-standing”. I’m not sure how that happens— perhaps Model::supportsTransactions?
yes - eventually models via their graph need to know about datasets. We
could create some kind of "dataset of one" (actually there already is)
to unify the architecture.
A corner case is that a model can be in a general dataset but come from
another one so it's in two+ at once. At the moment that's only possible
for the general dataset; otherwise Dataset(Graph) == storage
Avoid model transactions though:
* It's "begin()" which needs transaction promotion (and a risk of system
aborts) and that isn't current supported anywhere.
* It confuses the message that real/persistent storage is datasets these
days.
* May suggest nested transactions. Dataset -> model.
* SPARQL may be across models and or create them
>
>> The code took a read lock - not that this matters because it is not the dataset lock -
>
> No, the code is model.enterCriticalSection(false), a write lock. That’s why this is an interesting case. Well, interesting to me, anyway. {grin} That call is exactly the one I would want to block, but it doesn’t, because ModelCom knows nothing of the underlying dataset.
>
My bad. API before enums were around!
> ---
> A. Soroka
> The University of Virginia Library
>
>> On Dec 1, 2015, at 5:48 PM, Andy Seaborne <an...@apache.org> wrote:
>>
>> JENA-957
>> And this is related to the discussion on JENA-624 currently.
>>
>> DatasetGraphInMemory is a special case where it's lock (which is MR+SW not MRSW) is used internally. It is not the model lock is part of the issue here.
>>
>> We can encourage the use of transactions.
>>
>> It may be possible to make the lock interact with transactions:
>>
>> Thought:
>> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
>>
>>
>> The gist example mixes transactional and non-tranactional use.
>>
>> At L56 it's outside the write transaction so autocommit applies. As the W transaction never commits, the implicit second W transaction hangs.
>>
>> The code took a read lock - not that this matters because it is not the dataset lock -
>>
>> With locks, if it's a read lock, then write access is not checked - it's up to the application to behave.
>>
>> http://jena.apache.org/documentation/notes/concurrency-howto.html
>>
>> If it were the dataset lock, then in this case "things go wrong" because the app is manipulating the lock the transaction mechanism is using. I hope that if L54 was ds.getLock().enterCriticalSection(write) then it would block. A read lock from the dataset will let the thread through as it is MR+SW and then the Graph:add complain (??)
>>
>>
>> Elsewhere, in TDB, with great care, it is possible to MRSW inside a transaction. The upper level APIs add the ThreadLocal control to keep the hidden object but there really is an explicit Transaction object (org.apache.jena.tdb.transaction.Transaction).
>>
>> Andy
>>
>>
>> On 30/11/15 18:19, A. Soroka wrote:
>>> I noticed the other day that the ModelCom locking behavior seems completely independent of the locking behavior of the dataset that may underlie a Model. I’m a little worried about that with the advent of DatasetGraphInMemory, because I think the “ergonomics” could be confusing. I have a test case up here:
>>>
>>> https://gist.github.com/ajs6f/3e87cd6f78ec3b4e27a1
>>>
>>> The problem is that when used without transactions, DatasetGraphInMemory tries to “do what the user meant” by auto-wrapping ::add or ::delete operations in transactions. But this leads to a case where a thread can get a model from a DatasetGraphInMemory-backed dataset, acquire the write lock for that model with no trouble, but then block trying to add or delete anything from it because the mutation is blocked by some other thread elsewhere that has the dataset lock. What would seem less surprising to me would be for the first thread to block waiting to get the model write lock.
>>>
>>> Does this seem too obscure? Is it worth trying to think about how ModelCom could respect the locking of an underlying Dataset, if one exists?
>>>
>>> ---
>>> A. Soroka
>>> The University of Virginia Library
>>>
>>
>
Re: Model locks and Dataset locks
Posted by "A. Soroka" <aj...@virginia.edu>.
> Thought:
> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
This is pretty much where I was going with my question. But I think it would require something going on in ModelCom that could distinguish between those instances from a dataset and those that are “free-standing”. I’m not sure how that happens— perhaps Model::supportsTransactions?
> The code took a read lock - not that this matters because it is not the dataset lock -
No, the code is model.enterCriticalSection(false), a write lock. That’s why this is an interesting case. Well, interesting to me, anyway. {grin} That call is exactly the one I would want to block, but it doesn’t, because ModelCom knows nothing of the underlying dataset.
---
A. Soroka
The University of Virginia Library
> On Dec 1, 2015, at 5:48 PM, Andy Seaborne <an...@apache.org> wrote:
>
> JENA-957
> And this is related to the discussion on JENA-624 currently.
>
> DatasetGraphInMemory is a special case where it's lock (which is MR+SW not MRSW) is used internally. It is not the model lock is part of the issue here.
>
> We can encourage the use of transactions.
>
> It may be possible to make the lock interact with transactions:
>
> Thought:
> We (now) always have a transactional setup even if its only DatasetGraphWithLock. So a model from a dataset could use the datassets lock if the lock translated enter/leave into begin/end or begin/commit. A free-standing retains its own MRSW lock.
>
>
> The gist example mixes transactional and non-tranactional use.
>
> At L56 it's outside the write transaction so autocommit applies. As the W transaction never commits, the implicit second W transaction hangs.
>
> The code took a read lock - not that this matters because it is not the dataset lock -
>
> With locks, if it's a read lock, then write access is not checked - it's up to the application to behave.
>
> http://jena.apache.org/documentation/notes/concurrency-howto.html
>
> If it were the dataset lock, then in this case "things go wrong" because the app is manipulating the lock the transaction mechanism is using. I hope that if L54 was ds.getLock().enterCriticalSection(write) then it would block. A read lock from the dataset will let the thread through as it is MR+SW and then the Graph:add complain (??)
>
>
> Elsewhere, in TDB, with great care, it is possible to MRSW inside a transaction. The upper level APIs add the ThreadLocal control to keep the hidden object but there really is an explicit Transaction object (org.apache.jena.tdb.transaction.Transaction).
>
> Andy
>
>
> On 30/11/15 18:19, A. Soroka wrote:
>> I noticed the other day that the ModelCom locking behavior seems completely independent of the locking behavior of the dataset that may underlie a Model. I’m a little worried about that with the advent of DatasetGraphInMemory, because I think the “ergonomics” could be confusing. I have a test case up here:
>>
>> https://gist.github.com/ajs6f/3e87cd6f78ec3b4e27a1
>>
>> The problem is that when used without transactions, DatasetGraphInMemory tries to “do what the user meant” by auto-wrapping ::add or ::delete operations in transactions. But this leads to a case where a thread can get a model from a DatasetGraphInMemory-backed dataset, acquire the write lock for that model with no trouble, but then block trying to add or delete anything from it because the mutation is blocked by some other thread elsewhere that has the dataset lock. What would seem less surprising to me would be for the first thread to block waiting to get the model write lock.
>>
>> Does this seem too obscure? Is it worth trying to think about how ModelCom could respect the locking of an underlying Dataset, if one exists?
>>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>
Re: Model locks and Dataset locks
Posted by Andy Seaborne <an...@apache.org>.
JENA-957
And this is related to the discussion on JENA-624 currently.
DatasetGraphInMemory is a special case where it's lock (which is MR+SW
not MRSW) is used internally. It is not the model lock is part of the
issue here.
We can encourage the use of transactions.
It may be possible to make the lock interact with transactions:
Thought:
We (now) always have a transactional setup even if its only
DatasetGraphWithLock. So a model from a dataset could use the datassets
lock if the lock translated enter/leave into begin/end or begin/commit.
A free-standing retains its own MRSW lock.
The gist example mixes transactional and non-tranactional use.
At L56 it's outside the write transaction so autocommit applies. As the
W transaction never commits, the implicit second W transaction hangs.
The code took a read lock - not that this matters because it is not the
dataset lock -
With locks, if it's a read lock, then write access is not checked - it's
up to the application to behave.
http://jena.apache.org/documentation/notes/concurrency-howto.html
If it were the dataset lock, then in this case "things go wrong" because
the app is manipulating the lock the transaction mechanism is using. I
hope that if L54 was ds.getLock().enterCriticalSection(write) then it
would block. A read lock from the dataset will let the thread through
as it is MR+SW and then the Graph:add complain (??)
Elsewhere, in TDB, with great care, it is possible to MRSW inside a
transaction. The upper level APIs add the ThreadLocal control to keep
the hidden object but there really is an explicit Transaction object
(org.apache.jena.tdb.transaction.Transaction).
Andy
On 30/11/15 18:19, A. Soroka wrote:
> I noticed the other day that the ModelCom locking behavior seems completely independent of the locking behavior of the dataset that may underlie a Model. I’m a little worried about that with the advent of DatasetGraphInMemory, because I think the “ergonomics” could be confusing. I have a test case up here:
>
> https://gist.github.com/ajs6f/3e87cd6f78ec3b4e27a1
>
> The problem is that when used without transactions, DatasetGraphInMemory tries to “do what the user meant” by auto-wrapping ::add or ::delete operations in transactions. But this leads to a case where a thread can get a model from a DatasetGraphInMemory-backed dataset, acquire the write lock for that model with no trouble, but then block trying to add or delete anything from it because the mutation is blocked by some other thread elsewhere that has the dataset lock. What would seem less surprising to me would be for the first thread to block waiting to get the model write lock.
>
> Does this seem too obscure? Is it worth trying to think about how ModelCom could respect the locking of an underlying Dataset, if one exists?
>
> ---
> A. Soroka
> The University of Virginia Library
>