Iterators

[Andy Seaborne <an...@apache.org>]

Re: unclosed iterator in ResourceImpl.getPropertyResourceValue()
Re: MultiUnion multiGraphFind - closing iterators after exception

Jeremy,

Could you give some background?  closing of iterators has always been a 
bit tricky (because the reality is enough application code doesn't that 
we say all we like about "must close unfinished" - it is not going to 
happen).

In what circumstances does it matter to your code that unfinished 
iterators do get cleanly closed?  What's the environment?

	Andy

Re: Iterators

[Andy Seaborne <an...@apache.org>]

On 16/08/12 21:28, Jeremy Carroll wrote:
>
> I have written a locking graph that wraps a delegate and locks
> operations appropriately.

Unrelated, curiosity questions:

1/ So the triples are safe but the data model is not (e.g. adding two 
triples which are expected to be there together).  Are your unsafe 
readers just very careful.

2/ Is this noticeably expensive? Or is it that a non-issue for your 
application situation?

> Since unclosed iterators were obviously going to be an issue I do some
> GC tricks with a callback on a reference queue after the iterator is
> garbage collected to verify that close is in fact called, and if not to
> break the read lock that is still held.
> Using an optional (and very slow) flag to create an exception when the
> iterator is constructed I have detected many unclosed iterators in our
> own code, and I think those two in the Jena code base.
>
> While a DB graph is perhaps less catastrophic in its behavior on
> unclosed iterators, it is nevertheless an issue.
>
> Jeremy

I'm afraid the TDB code is much more precise than Java about checking. 
Java collections don't always CCME.

With (TDB) transactions, prior iterators are a non-issue because they 
are supported.

This effect can be achieved with any graphs using a binary union graph.

A wrapper graph has a reference to a binary union graph.

The left side of the union is for updates, the right side is the 
previous state of the graph.  It may be the base data or itself a union 
graph of an earlier writer.

Write back the changes into the base graph when the writer finished and 
all readers have finished.

Boundaries make that easier (.begin-> .commit or .end) but closing 
iterators is enough.

TDB does run in memory as well.  It was originally for testing (it is 
really running over RAM disks so not hugely performant).

If I had time (!!), a new DatasetGraph for memory with transactions 
would be fun to write.

	Andy

	Andy

Re: Iterators

[Jeremy Carroll <je...@topquadrant.com>]

I have written a locking graph that wraps a delegate and locks 
operations appropriately.

Since unclosed iterators were obviously going to be an issue I do some 
GC tricks with a callback on a reference queue after the iterator is 
garbage collected to verify that close is in fact called, and if not to 
break the read lock that is still held.
Using an optional (and very slow) flag to create an exception when the 
iterator is constructed I have detected many unclosed iterators in our 
own code, and I think those two in the Jena code base.

While a DB graph is perhaps less catastrophic in its behavior on 
unclosed iterators, it is nevertheless an issue.

Jeremy



On 8/16/2012 8:11 AM, Andy Seaborne wrote:
> Re: unclosed iterator in ResourceImpl.getPropertyResourceValue()
> Re: MultiUnion multiGraphFind - closing iterators after exception
>
> Jeremy,
>
> Could you give some background?  closing of iterators has always been 
> a bit tricky (because the reality is enough application code doesn't 
> that we say all we like about "must close unfinished" - it is not 
> going to happen).
>
> In what circumstances does it matter to your code that unfinished 
> iterators do get cleanly closed?  What's the environment?
>
>     Andy