You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Yonik Seeley <yo...@lucidimagination.com> on 2009/08/13 16:29:46 UTC
attribute thoughts
I'm liking the new attribute based analysis (in conjunction with
reusability), but I'm running into some questions...
Is it valid for tokenizers or token filters add new attributes after
their constructor (after they have processed some tokens)?
Should restoreState() be able to add attributes (it currently throws
an exception)? If not, does that mean that it's not supported/advised
to use state across different TokenStreams?
We've previously seen that the native java clone() can be much slower
than implementing it ourselves in Java. Should we have our own
clone() method on Attribute? Or just implement clone() ourselves and
require that subclasses override if needed? This is inner-loop
per-token stuff, and a single captureState() will invoke many clone
operations (6 attributes make up the legacy Token object).
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Michael Busch <bu...@gmail.com>.
On 8/18/09 5:44 PM, Yonik Seeley wrote:
> On Tue, Aug 18, 2009 at 8:01 PM, Michael Busch<bu...@gmail.com> wrote:
>
>> On 8/14/09 9:23 AM, Yonik Seeley wrote:
>>
>>> Another thing I was wondering about was the opacity of State - one
>>> can't inspect or change the attributes w/o restoring it first.
>>> Undesirable limitation, or feature allowing more flexible state
>>> implementations?
>>>
>>>
>> Excellent point! This limitation is currently there to discourage changing
>> values of
>> a state, because that would be rather inefficient: you'd have to lookup the
>> attribute(s)
>> of each state you want to change.
>>
> Yeah, but restoring state also involves looking up attributes... so if
> one is going to skip around a cached list of States (tokens), it can
> still be more efficient to not restore each.
>
Right, for this the proposed StateContainer should help.
What I meant was that often caching of tokens is not necessary at all.
There were two
filters in contrib (ShingleFilter and another one I can't recall right
now) that did a lot
of token caching; I rewrote them in a way so that they do most of their
work on the
fly, without cloning/caching.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Aug 18, 2009 at 8:01 PM, Michael Busch<bu...@gmail.com> wrote:
> On 8/14/09 9:23 AM, Yonik Seeley wrote:
>> Another thing I was wondering about was the opacity of State - one
>> can't inspect or change the attributes w/o restoring it first.
>> Undesirable limitation, or feature allowing more flexible state
>> implementations?
>>
>
> Excellent point! This limitation is currently there to discourage changing
> values of
> a state, because that would be rather inefficient: you'd have to lookup the
> attribute(s)
> of each state you want to change.
Yeah, but restoring state also involves looking up attributes... so if
one is going to skip around a cached list of States (tokens), it can
still be more efficient to not restore each.
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Michael Busch <bu...@gmail.com>.
On 8/14/09 9:23 AM, Yonik Seeley wrote:
> On Thu, Aug 13, 2009 at 4:32 PM, Michael Busch<bu...@gmail.com> wrote:
>
>> On 8/13/09 7:29 AM, Yonik Seeley wrote:
>>
>>> I'm liking the new attribute based analysis (in conjunction with
>>> reusability), but I'm running into some questions...
>>>
>>> Is it valid for tokenizers or token filters add new attributes after
>>> their constructor (after they have processed some tokens)?
>>>
>>>
>> At the moment we're saying in the javadocs of TokenStream that all
>> Attributes should be
>> added up front.
>>
> Hmmm, OK... in which case, token producers using restoreState() would
> not have to call clearAttributes() first.
>
>
>> We could change these semantics. I had some thoughts about
>> it in the original
>> JIRA issue (LUCENE-1422).
>>
> Apologies if I'm rehashing anything - it's hard to keep up with some
> of those monster (high volume) issues.
>
>
>> So back to your question if we should allow restoreState() to add attributes
>> and use a state across different AttributeSources: the complication is that we can only
>> allow that if the different AttributeSource were filled using the same AttributeFactory, otherwise
>> different AtttributeImpls could be in the sources and the copying wouldn't
>> work anymore.
>>
> Hmmm, so perhaps just an assertion that the factories are equal... and
> documentation saying that moving state from one stream to the other
> requires identical factories? Anyway, I don't currently have a use
> case for this... I was just wondering.
>
Yes that should work. We basically have such an assertion in
TeeSinkTokenFilter:
public void addSinkTokenStream(final SinkTokenStream sink) {
// check that sink has correct factory
if (!this.getAttributeFactory().equals(sink.getAttributeFactory())) {
throw new IllegalArgumentException("The supplied sink is not
compatible to this tee");
}
So I agree, we should just do the same in restoreState().
> Another thing I was wondering about was the opacity of State - one
> can't inspect or change the attributes w/o restoring it first.
> Undesirable limitation, or feature allowing more flexible state
> implementations?
>
>
Excellent point! This limitation is currently there to discourage
changing values of
a state, because that would be rather inefficient: you'd have to lookup
the attribute(s)
of each state you want to change. We could write a StateContainer, which
has an API
to access states in an efficient way (iterator, random access), using
delegation.
When I changed the contrib TokenStreams this limitation was somewhat
annoying
for some streams - but in all cases it was possible to implement the
streams far more
efficient by avoiding excessive caching. (except ShingleMatrixFilter, I
gave up eventually,
not knowing that code at all).
So I agree we should come up with a good API here for convenience, but
mention
in the javadocs that it should only be used carefully.
> -Yonik
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Aug 13, 2009 at 4:32 PM, Michael Busch<bu...@gmail.com> wrote:
> On 8/13/09 7:29 AM, Yonik Seeley wrote:
>>
>> I'm liking the new attribute based analysis (in conjunction with
>> reusability), but I'm running into some questions...
>>
>> Is it valid for tokenizers or token filters add new attributes after
>> their constructor (after they have processed some tokens)?
>>
> At the moment we're saying in the javadocs of TokenStream that all
> Attributes should be
> added up front.
Hmmm, OK... in which case, token producers using restoreState() would
not have to call clearAttributes() first.
> We could change these semantics. I had some thoughts about
> it in the original
> JIRA issue (LUCENE-1422).
Apologies if I'm rehashing anything - it's hard to keep up with some
of those monster (high volume) issues.
> So back to your question if we should allow restoreState() to add attributes
> and use a state across different AttributeSources: the complication is that we can only
> allow that if the different AttributeSource were filled using the same AttributeFactory, otherwise
> different AtttributeImpls could be in the sources and the copying wouldn't
> work anymore.
Hmmm, so perhaps just an assertion that the factories are equal... and
documentation saying that moving state from one stream to the other
requires identical factories? Anyway, I don't currently have a use
case for this... I was just wondering.
Another thing I was wondering about was the opacity of State - one
can't inspect or change the attributes w/o restoring it first.
Undesirable limitation, or feature allowing more flexible state
implementations?
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Michael Busch <bu...@gmail.com>.
On 8/13/09 7:29 AM, Yonik Seeley wrote:
> I'm liking the new attribute based analysis (in conjunction with
> reusability), but I'm running into some questions...
>
> Is it valid for tokenizers or token filters add new attributes after
> their constructor (after they have processed some tokens)?
>
>
At the moment we're saying in the javadocs of TokenStream that all
Attributes should be
added up front. We could change these semantics. I had some thoughts
about it in the original
JIRA issue (LUCENE-1422).
> Should restoreState() be able to add attributes (it currently throws
> an exception)? If not, does that mean that it's not supported/advised
> to use state across different TokenStreams?
>
>
See answer below.
> We've previously seen that the native java clone() can be much slower
> than implementing it ourselves in Java. Should we have our own
> clone() method on Attribute? Or just implement clone() ourselves and
> require that subclasses override if needed? This is inner-loop
> per-token stuff, and a single captureState() will invoke many clone
> operations (6 attributes make up the legacy Token object).
>
>
Improving the cloning performance was actually the main reason for
LUCENE-1693.
It separates the Attribute interfaces from the actual implementation,
and as you probably
know Token now implements all token attributes. So in a TokenStream
chain which does
cloning (e.g. with a TeeSinkTokenFilter or CachingTokenFilter) one could
use a different
AttributeFactory to get much better performance.
The AttributeSource builds internally a simple linked list (State),
which captureState()
clones then by calling the clone() method of the AttributeImpls. Using
the linked list approach
performed best for me. We could change the implementations of the
clone() methods of
the AttributeImpls or even add our own clone method if performance would
improve.
The nice thing about LUCENE-1693 is that if cloning performance is
really crucial for your
usecase you can simply implement a class that only implements the token
attributes you need.
Often term, positionIncrement and offset is enough. Then the object to
be cloned is smaller.
Ideally it'd be cool if we could synthesize a class automatically during
runtime that implements
all Attribute interfaces in use, but I think with java you can only do
that if you add a special
jar from the JDK to the classpath.
So back to your question if we should allow restoreState() to add
attributes and use a state
across different AttributeSources: the complication is that we can only
allow that if
the different AttributeSource were filled using the same
AttributeFactory, otherwise
different AtttributeImpls could be in the sources and the copying
wouldn't work anymore.
I didn't find a good (efficient) way of doing the cloning/copying per
Attribute interface yet,
which I did it this way. I'll try to think about if a bit more.... maybe
you have an idea?!
Michael
> -Yonik
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: attribute thoughts
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 13, 2009, at 10:29 AM, Yonik Seeley wrote:
> I'm liking the new attribute based analysis (in conjunction with
> reusability), but I'm running into some questions...
>
> Is it valid for tokenizers or token filters add new attributes after
> their constructor (after they have processed some tokens)?
I don't know if it explicitly prohibits it, but I believe the docs
discourage it for performance reasons.
>
> Should restoreState() be able to add attributes (it currently throws
> an exception)? If not, does that mean that it's not supported/advised
> to use state across different TokenStreams?
Not sure
>
> We've previously seen that the native java clone() can be much slower
> than implementing it ourselves in Java. Should we have our own
> clone() method on Attribute? Or just implement clone() ourselves and
> require that subclasses override if needed? This is inner-loop
> per-token stuff, and a single captureState() will invoke many clone
> operations (6 attributes make up the legacy Token object).
Sounds reasonable, but I haven't benchmarked.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org