You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Benson Margulies <bi...@gmail.com> on 2014/01/07 20:50:08 UTC
How is incrementToken supposed to detect the lack of reset()?
In 4.6.0, org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
fails if incrementToken fails to throw if there's a missing reset.
How am I supposed to organize this in a Tokenizer? A quick look at
CharTokenizer did not reveal any code for the purpose.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Benson Margulies <be...@basistech.com>.
I'm not in the delegate business, just a straight subclass. So I think they
are complementary. Gimme your github identity, and you are, as far as I am
concerned, more than welcome to add a section on delegates.
On Wed, Jan 8, 2014 at 7:38 AM, Mindaugas Žakšauskas <mi...@gmail.com>wrote:
> Hi,
>
> Sure, why not - I'm just not sure if my approach (of setting reader in
> reset()) is preferred over yours (using this.input instead of input in
> ctor)? Or are they both equally good?
>
> m.
>
> On Wed, Jan 8, 2014 at 12:18 PM, Benson Margulies <be...@basistech.com>
> wrote:
> > If you'd like to join in on the doc, see
> > https://github.com/apache/lucene-solr/pull/14/files. I'd be happy to
> grant
> > you access to push to my fork.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Mindaugas Žakšauskas <mi...@gmail.com>.
Hi,
Sure, why not - I'm just not sure if my approach (of setting reader in
reset()) is preferred over yours (using this.input instead of input in
ctor)? Or are they both equally good?
m.
On Wed, Jan 8, 2014 at 12:18 PM, Benson Margulies <be...@basistech.com> wrote:
> If you'd like to join in on the doc, see
> https://github.com/apache/lucene-solr/pull/14/files. I'd be happy to grant
> you access to push to my fork.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Benson Margulies <be...@basistech.com>.
If you'd like to join in on the doc, see
https://github.com/apache/lucene-solr/pull/14/files. I'd be happy to grant
you access to push to my fork.
On Wed, Jan 8, 2014 at 5:37 AM, Mindaugas Žakšauskas <mi...@gmail.com>wrote:
> Just for the interest, I had a similar problem too as well as other
> people [1]. In my project, I am extending the Tokenizer class and have
> another tokenizer (e.g. ClassicTokenizer) as a delegate.
> Unfortunately, properly overriding all public/protected methods is
> *not* enough, e.g.:
>
> public void reset() throws IOException {
> super.reset();
> delegate.reset();
> }
>
> I was still getting the exception of broken read()/close() contract.
> Half day and *lots* of debugging later, I realized that exception is
> only thrown when indexing second document only as the delegate reader
> internally gets replaced with ILLEGAL_STATE_READER after .close() is
> called. My solution to this problem was to make the reset() method
> like this:
>
> public void reset() throws IOException {
> super.reset();
> delegate.setReader(input);
> delegate.reset();
> }
>
> Another thing worth mentioning is that it's crucial to have
> super.method() before delegate.method() in all overridden methods.
> Would be nice if all of this was somewhere in the Tokenizer Javadoc,
> or even nicer if the base class was designed with delegation in mind
> (Effective Java (2nd edition), Item 16).
>
> Hope this helps somebody.
>
> [1]
> http://stackoverflow.com/questions/20624339/having-trouble-rereading-a-lucene-tokenstream/20630673#20630673
>
> Regards,
> Mindaugas
>
> On Tue, Jan 7, 2014 at 9:45 PM, Benson Margulies <be...@basistech.com>
> wrote:
> > Yes I Do.
> >
> >
> > On Tue, Jan 7, 2014 at 3:59 PM, Robert Muir <rc...@gmail.com> wrote:
> >
> >> Benson, do you want to open an issue to fix this constructor to not
> >> take Reader? (there might be one already, but lets make a new one).
> >>
> >> These things are supposed to be reused, and have setReader for that
> >> purpose. i think its confusing and contributes to bugs that you have
> >> to have logic in e.g. the ctor THEN ALSO in reset().
> >>
> >> if someone does it correctly in the ctor, but they only test "one
> >> time", they might think everything is working..
> >>
> >> On Tue, Jan 7, 2014 at 3:23 PM, Benson Margulies <be...@basistech.com>
> >> wrote:
> >> > For the record of other people who implement tokenizers:
> >> >
> >> > Say that your tokenizer has a constructor, like:
> >> >
> >> > public MyTokenizer(Reader reader, ....) {
> >> > super(reader);
> >> > myWrappedInputDevice = new MyWrappedInputDevice(reader);
> >> > }
> >> >
> >> > Not a good idea. Tokenizer carefully manages the data flow from the
> >> > constructor arg to the 'input' field. The correct form is:
> >> >
> >> > public MyTokenizer(Reader reader, ....) {
> >> > super(reader);
> >> > myWrappedInputDevice = new MyWrappedInputDevice(this.input);
> >> > }
> >> >
> >> >
> >> >
> >> > On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rc...@gmail.com> wrote:
> >> >
> >> >> See Tokenizer.java for the state machine logic. In general you should
> >> >> not have to do anything if the tokenizer is well-behaved (e.g. close
> >> >> calls super.close() and so on).
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <
> bimargulies@gmail.com
> >> >
> >> >> wrote:
> >> >> > In 4.6.0,
> >> >>
> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
> >> >> >
> >> >> > fails if incrementToken fails to throw if there's a missing reset.
> >> >> >
> >> >> > How am I supposed to organize this in a Tokenizer? A quick look at
> >> >> > CharTokenizer did not reveal any code for the purpose.
> >> >> >
> >> >> >
> ---------------------------------------------------------------------
> >> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >> >
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>
> >> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Mindaugas Žakšauskas <mi...@gmail.com>.
Just for the interest, I had a similar problem too as well as other
people [1]. In my project, I am extending the Tokenizer class and have
another tokenizer (e.g. ClassicTokenizer) as a delegate.
Unfortunately, properly overriding all public/protected methods is
*not* enough, e.g.:
public void reset() throws IOException {
super.reset();
delegate.reset();
}
I was still getting the exception of broken read()/close() contract.
Half day and *lots* of debugging later, I realized that exception is
only thrown when indexing second document only as the delegate reader
internally gets replaced with ILLEGAL_STATE_READER after .close() is
called. My solution to this problem was to make the reset() method
like this:
public void reset() throws IOException {
super.reset();
delegate.setReader(input);
delegate.reset();
}
Another thing worth mentioning is that it's crucial to have
super.method() before delegate.method() in all overridden methods.
Would be nice if all of this was somewhere in the Tokenizer Javadoc,
or even nicer if the base class was designed with delegation in mind
(Effective Java (2nd edition), Item 16).
Hope this helps somebody.
[1] http://stackoverflow.com/questions/20624339/having-trouble-rereading-a-lucene-tokenstream/20630673#20630673
Regards,
Mindaugas
On Tue, Jan 7, 2014 at 9:45 PM, Benson Margulies <be...@basistech.com> wrote:
> Yes I Do.
>
>
> On Tue, Jan 7, 2014 at 3:59 PM, Robert Muir <rc...@gmail.com> wrote:
>
>> Benson, do you want to open an issue to fix this constructor to not
>> take Reader? (there might be one already, but lets make a new one).
>>
>> These things are supposed to be reused, and have setReader for that
>> purpose. i think its confusing and contributes to bugs that you have
>> to have logic in e.g. the ctor THEN ALSO in reset().
>>
>> if someone does it correctly in the ctor, but they only test "one
>> time", they might think everything is working..
>>
>> On Tue, Jan 7, 2014 at 3:23 PM, Benson Margulies <be...@basistech.com>
>> wrote:
>> > For the record of other people who implement tokenizers:
>> >
>> > Say that your tokenizer has a constructor, like:
>> >
>> > public MyTokenizer(Reader reader, ....) {
>> > super(reader);
>> > myWrappedInputDevice = new MyWrappedInputDevice(reader);
>> > }
>> >
>> > Not a good idea. Tokenizer carefully manages the data flow from the
>> > constructor arg to the 'input' field. The correct form is:
>> >
>> > public MyTokenizer(Reader reader, ....) {
>> > super(reader);
>> > myWrappedInputDevice = new MyWrappedInputDevice(this.input);
>> > }
>> >
>> >
>> >
>> > On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rc...@gmail.com> wrote:
>> >
>> >> See Tokenizer.java for the state machine logic. In general you should
>> >> not have to do anything if the tokenizer is well-behaved (e.g. close
>> >> calls super.close() and so on).
>> >>
>> >>
>> >>
>> >> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bimargulies@gmail.com
>> >
>> >> wrote:
>> >> > In 4.6.0,
>> >> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
>> >> >
>> >> > fails if incrementToken fails to throw if there's a missing reset.
>> >> >
>> >> > How am I supposed to organize this in a Tokenizer? A quick look at
>> >> > CharTokenizer did not reveal any code for the purpose.
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Benson Margulies <be...@basistech.com>.
Yes I Do.
On Tue, Jan 7, 2014 at 3:59 PM, Robert Muir <rc...@gmail.com> wrote:
> Benson, do you want to open an issue to fix this constructor to not
> take Reader? (there might be one already, but lets make a new one).
>
> These things are supposed to be reused, and have setReader for that
> purpose. i think its confusing and contributes to bugs that you have
> to have logic in e.g. the ctor THEN ALSO in reset().
>
> if someone does it correctly in the ctor, but they only test "one
> time", they might think everything is working..
>
> On Tue, Jan 7, 2014 at 3:23 PM, Benson Margulies <be...@basistech.com>
> wrote:
> > For the record of other people who implement tokenizers:
> >
> > Say that your tokenizer has a constructor, like:
> >
> > public MyTokenizer(Reader reader, ....) {
> > super(reader);
> > myWrappedInputDevice = new MyWrappedInputDevice(reader);
> > }
> >
> > Not a good idea. Tokenizer carefully manages the data flow from the
> > constructor arg to the 'input' field. The correct form is:
> >
> > public MyTokenizer(Reader reader, ....) {
> > super(reader);
> > myWrappedInputDevice = new MyWrappedInputDevice(this.input);
> > }
> >
> >
> >
> > On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rc...@gmail.com> wrote:
> >
> >> See Tokenizer.java for the state machine logic. In general you should
> >> not have to do anything if the tokenizer is well-behaved (e.g. close
> >> calls super.close() and so on).
> >>
> >>
> >>
> >> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bimargulies@gmail.com
> >
> >> wrote:
> >> > In 4.6.0,
> >> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
> >> >
> >> > fails if incrementToken fails to throw if there's a missing reset.
> >> >
> >> > How am I supposed to organize this in a Tokenizer? A quick look at
> >> > CharTokenizer did not reveal any code for the purpose.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Robert Muir <rc...@gmail.com>.
Benson, do you want to open an issue to fix this constructor to not
take Reader? (there might be one already, but lets make a new one).
These things are supposed to be reused, and have setReader for that
purpose. i think its confusing and contributes to bugs that you have
to have logic in e.g. the ctor THEN ALSO in reset().
if someone does it correctly in the ctor, but they only test "one
time", they might think everything is working..
On Tue, Jan 7, 2014 at 3:23 PM, Benson Margulies <be...@basistech.com> wrote:
> For the record of other people who implement tokenizers:
>
> Say that your tokenizer has a constructor, like:
>
> public MyTokenizer(Reader reader, ....) {
> super(reader);
> myWrappedInputDevice = new MyWrappedInputDevice(reader);
> }
>
> Not a good idea. Tokenizer carefully manages the data flow from the
> constructor arg to the 'input' field. The correct form is:
>
> public MyTokenizer(Reader reader, ....) {
> super(reader);
> myWrappedInputDevice = new MyWrappedInputDevice(this.input);
> }
>
>
>
> On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rc...@gmail.com> wrote:
>
>> See Tokenizer.java for the state machine logic. In general you should
>> not have to do anything if the tokenizer is well-behaved (e.g. close
>> calls super.close() and so on).
>>
>>
>>
>> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bi...@gmail.com>
>> wrote:
>> > In 4.6.0,
>> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
>> >
>> > fails if incrementToken fails to throw if there's a missing reset.
>> >
>> > How am I supposed to organize this in a Tokenizer? A quick look at
>> > CharTokenizer did not reveal any code for the purpose.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Benson Margulies <be...@basistech.com>.
For the record of other people who implement tokenizers:
Say that your tokenizer has a constructor, like:
public MyTokenizer(Reader reader, ....) {
super(reader);
myWrappedInputDevice = new MyWrappedInputDevice(reader);
}
Not a good idea. Tokenizer carefully manages the data flow from the
constructor arg to the 'input' field. The correct form is:
public MyTokenizer(Reader reader, ....) {
super(reader);
myWrappedInputDevice = new MyWrappedInputDevice(this.input);
}
On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rc...@gmail.com> wrote:
> See Tokenizer.java for the state machine logic. In general you should
> not have to do anything if the tokenizer is well-behaved (e.g. close
> calls super.close() and so on).
>
>
>
> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bi...@gmail.com>
> wrote:
> > In 4.6.0,
> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
> >
> > fails if incrementToken fails to throw if there's a missing reset.
> >
> > How am I supposed to organize this in a Tokenizer? A quick look at
> > CharTokenizer did not reveal any code for the purpose.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: How is incrementToken supposed to detect the lack of reset()?
Posted by Robert Muir <rc...@gmail.com>.
See Tokenizer.java for the state machine logic. In general you should
not have to do anything if the tokenizer is well-behaved (e.g. close
calls super.close() and so on).
On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bi...@gmail.com> wrote:
> In 4.6.0, org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
>
> fails if incrementToken fails to throw if there's a missing reset.
>
> How am I supposed to organize this in a Tokenizer? A quick look at
> CharTokenizer did not reveal any code for the purpose.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org