You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by Marshall Schor <ms...@schor.com> on 2007/07/12 15:10:33 UTC
[Fwd: Re: Iterators: problem when using standard methods in combination
with moveTo*]
Thilo - is this "fixable" - so it just works as users expect?
-Marshall
-------- Original Message --------
Subject: Re: Iterators: problem when using standard methods in
combination with moveTo*
Date: Thu, 12 Jul 2007 13:33:31 +0200
From: Thilo Goetz <tw...@gmx.de>
Reply-To: uima-user@incubator.apache.org
To: uima-user@incubator.apache.org
References:
<07...@comcast.net>
<46...@schor.com> <46...@gmx.de>
<46...@dcs.shef.ac.uk> <46...@schor.com>
<46...@gmx.de> <46...@dcs.shef.ac.uk>
Hi Julien,
Julien Nioche wrote:
> Thilo and Marshall,
>
> Thanks for sharing the tip. Indeed it would be a good idea to add this
> little example to the documentation.
>
> A quick comment about the Iterator methods. I had a problem with the
> following piece of code:
>
> /while (wordFormIterator.hasNext()){
> WordForm wf = (WordForm)wordFormIterator.next();
> if (wf.getBegin()==token.getBegin() && wf.getEnd()==token.getEnd()){
> liste.add(wf);
> }
> else {
> // move back
> wordFormIterator.moveToPrevious();
> return liste;
> }
> }
> /
> The last element of the iterator was never accessible because
> /hasNext()/ returned false despite the fact that there WAS an element
> left in there. /moveToPrevious /had been previously called on this iterator.
>
> Should not /hasNext() /return true even if the cursor has been moved
> forward or backward within the iterator? Or is the use of the legacy
> methods (hasNext(), next()) incompatible with the /moveTo* /methods?
hm, I thought this was in our documentation, but couldn't find it myself.
You should not mix the use of next()/hasNext() with the methods defined
in the FSIterator interface. They do not work well together. If you use
the FSIterator APIs, you should use them exclusively. Sorry about that.
I'll add a comment to the javadocs.
>
> Thanks
>
> Julien
>> To be a bit more explicit, here's some code that will determine how
>> many tokens the longest sentence in the document contains. It's a
>> silly example, but it illustrates the concept. Maybe this should go
>> in the docs. Note: I have not actually run this code, it may not
>> work immediately ;-)
>>
>> CAS cas = ...;
>> Type sentenceType = cas.getTypeSystem().getType("yourSentenceTypeName");
>> Type tokenType = cas.getTypeSystem().getType("yourTokenTypeName");
>> FSIterator sentenceIt = cas.getAnnotationIndex(sentenceType).iterator();
>> AnnotationIndex tokenIndex = cas.getAnnotationIndex(tokenType);
>> FSIterator tokenIt;
>> int maxLen = 0;
>> int currentLen;
>> for (sentenceIt.moveToFirst(); sentenceIt.isValid(); sentenceIt.moveToNext()) {
>> tokenIt = tokenIndex.subiterator((AnnotationFS) sentenceIt.get());
>> currentLen = 0;
>> for (tokenIt.moveToFirst(); tokenIt.isValid(); tokenIt.moveToNext()) {
>> ++currentLen;
>> }
>> maxLen = ((maxLen < currentLen) ? currentLen : maxLen);
>> }
>> System.out.println("Longest sentence contains " + maxLen + " tokens.");
>>
>> --Thilo
>>
>> Marshall Schor wrote:
>>
>>> Did you consider using subIterators? These are (briefly) described in
>>> section 4.7.4 of the Apache UIMA Reference book, and may include exactly
>>> what you're trying to get at - an interator over elements that are
>>> "contained" in the span of other elements.
>>>
>>> -Marshall
>>>
>>> Julien Nioche wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry if someone already asked the question.
>>>> Is there a direct way to obtain from a Cas all the annotations of a
>>>> given type located between two positions in the text? Something like
>>>> getContained(String type,int start,int end)?
>>>> I am trying to get all the Tokens contained within a specific
>>>> Sentence. I have used iterators for doing that and compared the offset
>>>> with those of the Sentence but it is a bit tedious. Have I missed
>>>> something obvious?
>>>>
>>>> Thanks
>>>>
>>>> Julien
>>>>
>>>>
>>>>
>
Re: [Fwd: Re: Iterators: problem when using standard methods in combination
with moveTo*]
Posted by Marshall Schor <ms...@schor.com>.
Those arguments sound convincing to me. I guess I would only prefer the
docs were less mysterious,
and in addition to saying "not to mix the two styles", they would
state the reasons, as you have here :-)
-Marshall
Thilo Goetz wrote:
> Marshall Schor wrote:
>
>> It seems to me the only object of confusion arises when users use
>> next() to get an element and move to the next element, and then use
>> moveToPrevious, which as you say, may or may not work if the iterator
>> ended up "invalid" because the next() moved it past the last element.
>>
>> So the only "improvement" I would think is wanted is to make the moveTo
>> operations work reliably in these kinds of cases. Is that hard to do?
>>
>> -Marshall
>>
>
> What do you mean by "reliably"? Suppose I'm at the end of an iteration,
> and I call moveToNext() 3 more times. How many times do I need to call
> moveToPrevious() to make the iterator valid again? Does each iterator
> implementation need to track that kind of information?
>
> The problem is that this sort of thing is not part of the iterator
> contract, and we have quite a few iterator implementations, for various
> indexes etc. When an iterator !isValid(), all bets are off. You can
> call moveToFirst(), moveToLast(), or moveTo(FS) to reset it.
>
> hasNext()/next() were simply not designed for that kind of environment,
> which is why I originally implemented a different interface. I think
> that's why, for example, there's no previous() in that API. Suppose
> you were to call next(), then previous(). What would the expected
> result be? Would you expect to get the same element twice, or not?
> I personally have no intuition.
>
> So I'll stick with my earlier recommendation: don't mix the two styles.
> If you just need to iterate from start to end, by all means use the
> java.util.Iterator style. Else, use the FSIterator API.
>
> --Thilo
>
>
>
Re: [Fwd: Re: Iterators: problem when using standard methods in combination
with moveTo*]
Posted by Thilo Goetz <tw...@gmx.de>.
Marshall Schor wrote:
> It seems to me the only object of confusion arises when users use
> next() to get an element and move to the next element, and then use
> moveToPrevious, which as you say, may or may not work if the iterator
> ended up "invalid" because the next() moved it past the last element.
>
> So the only "improvement" I would think is wanted is to make the moveTo
> operations work reliably in these kinds of cases. Is that hard to do?
>
> -Marshall
What do you mean by "reliably"? Suppose I'm at the end of an iteration,
and I call moveToNext() 3 more times. How many times do I need to call
moveToPrevious() to make the iterator valid again? Does each iterator
implementation need to track that kind of information?
The problem is that this sort of thing is not part of the iterator
contract, and we have quite a few iterator implementations, for various
indexes etc. When an iterator !isValid(), all bets are off. You can
call moveToFirst(), moveToLast(), or moveTo(FS) to reset it.
hasNext()/next() were simply not designed for that kind of environment,
which is why I originally implemented a different interface. I think
that's why, for example, there's no previous() in that API. Suppose
you were to call next(), then previous(). What would the expected
result be? Would you expect to get the same element twice, or not?
I personally have no intuition.
So I'll stick with my earlier recommendation: don't mix the two styles.
If you just need to iterate from start to end, by all means use the
java.util.Iterator style. Else, use the FSIterator API.
--Thilo
Re: [Fwd: Re: Iterators: problem when using standard methods in combination
with moveTo*]
Posted by Marshall Schor <ms...@schor.com>.
It seems to me the only object of confusion arises when users use
next() to get an element and move to the next element, and then use
moveToPrevious, which as you say, may or may not work if the iterator
ended up "invalid" because the next() moved it past the last element.
So the only "improvement" I would think is wanted is to make the moveTo
operations work reliably in these kinds of cases. Is that hard to do?
-Marshall
Thilo Goetz wrote:
> What is the expected behavior? Here's the impl:
>
> public boolean hasNext() {
> return isValid();
> }
>
> public Object next() {
> Object result = get();
> moveToNext();
> return result;
> }
>
> Perfectly reasonable, but has some consequences
> that may not be obvious. For example, when you
> do
>
> FS fs1 = it.next();
> FS fs2 = it.get();
>
> then fs1 != fs2. Is that intuitive? I don't know.
> Is it fixable? Not easily, no.
>
> We also have this more subtle behavior:
>
> FS fs1 = it.next();
> it.moveToPrevious();
> FS fs2 = it.next();
>
> The last line may throw a NoSuchElementException. Why?
> Because the first line may invalidate the iterator, and
> then moveToPrevious() will not normally make the iterator
> valid again (sometimes it will, depending on the iterator
> implementation).
>
> So in terms of what is reasonable, the iterators behave
> as expected. Still, because the interacations are so
> subtle, it is not a good idea to mix the paradigms. I
> never do, even though I think I understand what's going
> on.
>
> I'm pretty sure I've documented this before. I don't know
> where that text went. Maybe I dreamed it.
>
> --Thilo
>
> Marshall Schor wrote:
>
>> Thilo - is this "fixable" - so it just works as users expect?
>>
>> -Marshall
>>
>> -------- Original Message --------
>> Subject: Re: Iterators: problem when using standard methods in
>> combination with moveTo*
>> Date: Thu, 12 Jul 2007 13:33:31 +0200
>> From: Thilo Goetz <tw...@gmx.de>
>> Reply-To: uima-user@incubator.apache.org
>> To: uima-user@incubator.apache.org
>> References:
>> <07...@comcast.net>
>>
>> <46...@schor.com> <46...@gmx.de>
>> <46...@dcs.shef.ac.uk> <46...@schor.com>
>> <46...@gmx.de> <46...@dcs.shef.ac.uk>
>>
>>
>>
>> Hi Julien,
>>
>> Julien Nioche wrote:
>>
>>> Thilo and Marshall,
>>>
>>> Thanks for sharing the tip. Indeed it would be a good idea to add this
>>> little example to the documentation.
>>>
>>> A quick comment about the Iterator methods. I had a problem with the
>>> following piece of code:
>>>
>>> /while (wordFormIterator.hasNext()){
>>> WordForm wf = (WordForm)wordFormIterator.next();
>>> if (wf.getBegin()==token.getBegin() && wf.getEnd()==token.getEnd()){
>>> liste.add(wf);
>>> }
>>> else {
>>> // move back
>>> wordFormIterator.moveToPrevious();
>>> return liste;
>>> }
>>> }
>>> /
>>> The last element of the iterator was never accessible because
>>> /hasNext()/ returned false despite the fact that there WAS an element
>>> left in there. /moveToPrevious /had been previously called on this
>>> iterator.
>>>
>>> Should not /hasNext() /return true even if the cursor has been moved
>>> forward or backward within the iterator? Or is the use of the legacy
>>> methods (hasNext(), next()) incompatible with the /moveTo* /methods?
>>>
>> hm, I thought this was in our documentation, but couldn't find it myself.
>> You should not mix the use of next()/hasNext() with the methods defined
>> in the FSIterator interface. They do not work well together. If you use
>> the FSIterator APIs, you should use them exclusively. Sorry about that.
>> I'll add a comment to the javadocs.
>>
>>
>>> Thanks
>>>
>>> Julien
>>>
>>>> To be a bit more explicit, here's some code that will determine how
>>>> many tokens the longest sentence in the document contains. It's a
>>>> silly example, but it illustrates the concept. Maybe this should go
>>>> in the docs. Note: I have not actually run this code, it may not
>>>> work immediately ;-)
>>>>
>>>> CAS cas = ...;
>>>> Type sentenceType =
>>>> cas.getTypeSystem().getType("yourSentenceTypeName");
>>>> Type tokenType = cas.getTypeSystem().getType("yourTokenTypeName");
>>>> FSIterator sentenceIt =
>>>> cas.getAnnotationIndex(sentenceType).iterator();
>>>> AnnotationIndex tokenIndex = cas.getAnnotationIndex(tokenType);
>>>> FSIterator tokenIt;
>>>> int maxLen = 0;
>>>> int currentLen;
>>>> for (sentenceIt.moveToFirst(); sentenceIt.isValid();
>>>> sentenceIt.moveToNext()) {
>>>> tokenIt = tokenIndex.subiterator((AnnotationFS) sentenceIt.get());
>>>> currentLen = 0;
>>>> for (tokenIt.moveToFirst(); tokenIt.isValid();
>>>> tokenIt.moveToNext()) {
>>>> ++currentLen;
>>>> }
>>>> maxLen = ((maxLen < currentLen) ? currentLen : maxLen);
>>>> }
>>>> System.out.println("Longest sentence contains " + maxLen + "
>>>> tokens.");
>>>>
>>>> --Thilo
>>>>
>>>> Marshall Schor wrote:
>>>>
>>>>
>>>>> Did you consider using subIterators? These are (briefly) described in
>>>>> section 4.7.4 of the Apache UIMA Reference book, and may include
>>>>> exactly
>>>>> what you're trying to get at - an interator over elements that are
>>>>> "contained" in the span of other elements.
>>>>>
>>>>> -Marshall
>>>>>
>>>>> Julien Nioche wrote:
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry if someone already asked the question.
>>>>>> Is there a direct way to obtain from a Cas all the annotations of a
>>>>>> given type located between two positions in the text? Something like
>>>>>> getContained(String type,int start,int end)?
>>>>>> I am trying to get all the Tokens contained within a specific
>>>>>> Sentence. I have used iterators for doing that and compared the offset
>>>>>> with those of the Sentence but it is a bit tedious. Have I missed
>>>>>> something obvious?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Julien
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>
>>
>
>
>
Re: [Fwd: Re: Iterators: problem when using standard methods in combination
with moveTo*]
Posted by Thilo Goetz <tw...@gmx.de>.
What is the expected behavior? Here's the impl:
public boolean hasNext() {
return isValid();
}
public Object next() {
Object result = get();
moveToNext();
return result;
}
Perfectly reasonable, but has some consequences
that may not be obvious. For example, when you
do
FS fs1 = it.next();
FS fs2 = it.get();
then fs1 != fs2. Is that intuitive? I don't know.
Is it fixable? Not easily, no.
We also have this more subtle behavior:
FS fs1 = it.next();
it.moveToPrevious();
FS fs2 = it.next();
The last line may throw a NoSuchElementException. Why?
Because the first line may invalidate the iterator, and
then moveToPrevious() will not normally make the iterator
valid again (sometimes it will, depending on the iterator
implementation).
So in terms of what is reasonable, the iterators behave
as expected. Still, because the interacations are so
subtle, it is not a good idea to mix the paradigms. I
never do, even though I think I understand what's going
on.
I'm pretty sure I've documented this before. I don't know
where that text went. Maybe I dreamed it.
--Thilo
Marshall Schor wrote:
> Thilo - is this "fixable" - so it just works as users expect?
>
> -Marshall
>
> -------- Original Message --------
> Subject: Re: Iterators: problem when using standard methods in
> combination with moveTo*
> Date: Thu, 12 Jul 2007 13:33:31 +0200
> From: Thilo Goetz <tw...@gmx.de>
> Reply-To: uima-user@incubator.apache.org
> To: uima-user@incubator.apache.org
> References:
> <07...@comcast.net>
>
> <46...@schor.com> <46...@gmx.de>
> <46...@dcs.shef.ac.uk> <46...@schor.com>
> <46...@gmx.de> <46...@dcs.shef.ac.uk>
>
>
>
> Hi Julien,
>
> Julien Nioche wrote:
>> Thilo and Marshall,
>>
>> Thanks for sharing the tip. Indeed it would be a good idea to add this
>> little example to the documentation.
>>
>> A quick comment about the Iterator methods. I had a problem with the
>> following piece of code:
>>
>> /while (wordFormIterator.hasNext()){
>> WordForm wf = (WordForm)wordFormIterator.next();
>> if (wf.getBegin()==token.getBegin() && wf.getEnd()==token.getEnd()){
>> liste.add(wf);
>> }
>> else {
>> // move back
>> wordFormIterator.moveToPrevious();
>> return liste;
>> }
>> }
>> /
>> The last element of the iterator was never accessible because
>> /hasNext()/ returned false despite the fact that there WAS an element
>> left in there. /moveToPrevious /had been previously called on this
>> iterator.
>>
>> Should not /hasNext() /return true even if the cursor has been moved
>> forward or backward within the iterator? Or is the use of the legacy
>> methods (hasNext(), next()) incompatible with the /moveTo* /methods?
>
> hm, I thought this was in our documentation, but couldn't find it myself.
> You should not mix the use of next()/hasNext() with the methods defined
> in the FSIterator interface. They do not work well together. If you use
> the FSIterator APIs, you should use them exclusively. Sorry about that.
> I'll add a comment to the javadocs.
>
>>
>> Thanks
>>
>> Julien
>>> To be a bit more explicit, here's some code that will determine how
>>> many tokens the longest sentence in the document contains. It's a
>>> silly example, but it illustrates the concept. Maybe this should go
>>> in the docs. Note: I have not actually run this code, it may not
>>> work immediately ;-)
>>>
>>> CAS cas = ...;
>>> Type sentenceType =
>>> cas.getTypeSystem().getType("yourSentenceTypeName");
>>> Type tokenType = cas.getTypeSystem().getType("yourTokenTypeName");
>>> FSIterator sentenceIt =
>>> cas.getAnnotationIndex(sentenceType).iterator();
>>> AnnotationIndex tokenIndex = cas.getAnnotationIndex(tokenType);
>>> FSIterator tokenIt;
>>> int maxLen = 0;
>>> int currentLen;
>>> for (sentenceIt.moveToFirst(); sentenceIt.isValid();
>>> sentenceIt.moveToNext()) {
>>> tokenIt = tokenIndex.subiterator((AnnotationFS) sentenceIt.get());
>>> currentLen = 0;
>>> for (tokenIt.moveToFirst(); tokenIt.isValid();
>>> tokenIt.moveToNext()) {
>>> ++currentLen;
>>> }
>>> maxLen = ((maxLen < currentLen) ? currentLen : maxLen);
>>> }
>>> System.out.println("Longest sentence contains " + maxLen + "
>>> tokens.");
>>>
>>> --Thilo
>>>
>>> Marshall Schor wrote:
>>>
>>>> Did you consider using subIterators? These are (briefly) described in
>>>> section 4.7.4 of the Apache UIMA Reference book, and may include
>>>> exactly
>>>> what you're trying to get at - an interator over elements that are
>>>> "contained" in the span of other elements.
>>>>
>>>> -Marshall
>>>>
>>>> Julien Nioche wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry if someone already asked the question.
>>>>> Is there a direct way to obtain from a Cas all the annotations of a
>>>>> given type located between two positions in the text? Something like
>>>>> getContained(String type,int start,int end)?
>>>>> I am trying to get all the Tokens contained within a specific
>>>>> Sentence. I have used iterators for doing that and compared the offset
>>>>> with those of the Sentence but it is a bit tedious. Have I missed
>>>>> something obvious?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Julien
>>>>>
>>>>>
>>>>>
>>
>
>
>