You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Busch <bu...@gmail.com> on 2008/10/12 01:33:42 UTC
TokenStream and Token APIs
Hi,
I've been thinking about making the TokenStream and Token APIs more
flexible. E. g. for fields that don't store positions, the Token doesn't
need to have a positionIncrement or a payload. With flexible indexing on
the other hand, people might want to add custom attributes to a Token
that a consumer in the indexing chain could use then.
Of course it is possible to extend Token, because it is not final, and
add additional attributes to it. But then consumers of the TokenStream
must downcast every instance of the Token object when they call
next(Token).
I was therefore thinking about a different TokenStream API:
public abstract class TokenStream {
public abstract boolean nextToken() throws IOException;
public abstract Token prototypeToken() throws IOException;
public void reset() throws IOException {}
public void close() throws IOException {}
}
Furthermore Token itself would only keep the termBuffer logic and we
could introduce different interfaces, like:
public interface PayloadAttribute {
/**
* Returns this Token's payload.
*/
public Payload getPayload();
/**
* Sets this Token's payload.
*/
public void setPayload(Payload payload);
}
public interface PositionIncrementAttribute {
/** Set the position increment. This determines the position of
* this token relative to the previous Token in a
* {@link TokenStream}, used in phrase searching.
*/
public void setPositionIncrement(int positionIncrement);
/** Returns the position increment of this Token.
* @see #setPositionIncrement
*/
public int getPositionIncrement();
}
A consumer, e. g. the DocumentsWriter, does not create a Token instance
itself anymore, but rather calls prototypeToken(). This method returns a
Token subclass which implements all desired *Attribute interfaces.
If a consumer is e. g. only interested in the positionIncrement and
Payload, it can consume the tokens like this:
public class Consumer {
public void consumeTokens(TokenStream ts) throws IOException {
Token token = ts.prototypeToken();
PayloadAttribute payloadSource = (PayloadAttribute) token;
PositionIncrementAttribute positionSource =
(PositionIncrementAttribute) token;
while (ts.nextToken()) {
char[] term = token.termBuffer();
int termLength = token.termLength();
int positionIncrement = positionSource.getPositionIncrement();
Payload payload = payloadSource.getPayload();
// do something with the term, positionIncrement and payload
}
}
}
Casting is now only done once after the prototype token was created. Now
if you want to add another consumer in the indexing chain and realize
that you want to add another attribute to the Token, then you don't have
to change this consumer. You only need to create another Token subclass
that implements the new attribute in addition to the previous ones and
can use it in the new consumer.
I haven't tried to implement this yet and maybe there are things I
haven't thought about (like caching TokenFilters). I'd like to get some
feedback about these APIs first to see if this makes sense?
Btw: if we think this (or another) approach to change these APIs makes
sense, then it would be good to change it for 3.0 when we can break
backwards compatibility. And then we should also rethink the
Fieldable/AbstractField/Field/FieldInfos APIs for 3.0 and flexible indexing!
-Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Michael McCandless <lu...@mikemccandless.com>.
This looks good!
One question on back compatibility: currently, TokenStream.nextToken
takes a Token arg in, and returns a Token back, such that the method
is encouraged but not required to use the passed-in Token as its
prototype.
You are adding a boolean nextToken() method, which then forces the
reuse (which I think is good) but you need to ensure older TokenStream
impls still work. I guess this amounts to a default implementation of
boolean nextToken() in the base TokenStream class.
Mike
Michael Busch wrote:
> Hi,
>
> I've been thinking about making the TokenStream and Token APIs more
> flexible. E. g. for fields that don't store positions, the Token
> doesn't need to have a positionIncrement or a payload. With flexible
> indexing on the other hand, people might want to add custom
> attributes to a Token that a consumer in the indexing chain could
> use then.
>
> Of course it is possible to extend Token, because it is not final,
> and add additional attributes to it. But then consumers of the
> TokenStream must downcast every instance of the Token object when
> they call next(Token).
>
> I was therefore thinking about a different TokenStream API:
>
> public abstract class TokenStream {
> public abstract boolean nextToken() throws IOException;
>
> public abstract Token prototypeToken() throws IOException;
>
> public void reset() throws IOException {}
>
> public void close() throws IOException {}
> }
>
> Furthermore Token itself would only keep the termBuffer logic and we
> could introduce different interfaces, like:
>
> public interface PayloadAttribute {
> /**
> * Returns this Token's payload.
> */
> public Payload getPayload();
>
> /**
> * Sets this Token's payload.
> */
> public void setPayload(Payload payload);
> }
>
> public interface PositionIncrementAttribute {
> /** Set the position increment. This determines the position of
> * this token relative to the previous Token in a
> * {@link TokenStream}, used in phrase searching.
> */
> public void setPositionIncrement(int positionIncrement);
>
> /** Returns the position increment of this Token.
> * @see #setPositionIncrement
> */
> public int getPositionIncrement();
> }
>
> A consumer, e. g. the DocumentsWriter, does not create a Token
> instance itself anymore, but rather calls prototypeToken(). This
> method returns a Token subclass which implements all desired
> *Attribute interfaces.
>
> If a consumer is e. g. only interested in the positionIncrement and
> Payload, it can consume the tokens like this:
>
> public class Consumer {
> public void consumeTokens(TokenStream ts) throws IOException {
> Token token = ts.prototypeToken();
>
> PayloadAttribute payloadSource = (PayloadAttribute) token;
> PositionIncrementAttribute positionSource =
> (PositionIncrementAttribute) token;
>
> while (ts.nextToken()) {
> char[] term = token.termBuffer();
> int termLength = token.termLength();
> int positionIncrement = positionSource.getPositionIncrement();
> Payload payload = payloadSource.getPayload();
>
> // do something with the term, positionIncrement and payload
> }
> }
> }
>
> Casting is now only done once after the prototype token was created.
> Now if you want to add another consumer in the indexing chain and
> realize that you want to add another attribute to the Token, then
> you don't have to change this consumer. You only need to create
> another Token subclass that implements the new attribute in addition
> to the previous ones and can use it in the new consumer.
>
> I haven't tried to implement this yet and maybe there are things I
> haven't thought about (like caching TokenFilters). I'd like to get
> some feedback about these APIs first to see if this makes sense?
>
> Btw: if we think this (or another) approach to change these APIs
> makes sense, then it would be good to change it for 3.0 when we can
> break backwards compatibility. And then we should also rethink the
> Fieldable/AbstractField/Field/FieldInfos APIs for 3.0 and flexible
> indexing!
>
> -Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Michael Busch <bu...@gmail.com>.
Mark Miller wrote:
> Grant Ingersoll wrote:
>>
>> On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
>>
>>> Grant Ingersoll wrote:
>>>>
>>>> Bear with me, b/c I'm not sure I'm following, but looking at
>>>> https://issues.apache.org/jira/browse/LUCENE-1422, I see at least 5
>>>> different implemented Attributes.
>>>>
>>>> So, let's say I add a 5 more attributes and now have a total of 10
>>>> attributes. Are you saying that I then would have, potentially, 10
>>>> different variables that all point to the token as in the code
>>>> snippet above where the casting takes place? Or would I just create
>>>> a single "Super" attribute that folds in all of my new attributes,
>>>> plus any other existing ones? Or, maybe, what I would do is create
>>>> the 5 new attributes and then 1 new attribute that extends all 10,
>>>> thus allowing me to use them individually, but saving me from
>>>> having to do a whole ton of casting in my Consumer.
>>> Potentially one consumer doing 10 things, but not likely right? I
>>> mean, things will stay logical as they are now, and rather than a
>>> super consumer doing everything, we will still have a chain of
>>> consumers each doing its own piece. So more likely, maybe something
>>> comes along every so often (another 5, over *much* time, say) and
>>> each time we add a Consumer that uses one or two TokenStream types.
>>> And then its just an implementation detail on whether you make a
>>> composite TokenStream - if you have added 10 new attributes and see
>>> it fit to make one consumer use them all, sure, make a composite,
>>> super type, but in my mind, the way its done in the example code is
>>> clearer/cleaner for a handful of TokenStream types. And even if you
>>> do make the composite,super type, its likely to just be a sugar
>>> wrapper anyway - the implementation for say, payload and positions,
>>> should probably be maintained in their own classes anyway.
>>
>> Well, there are 5 different attributes already, all of which are
>> commonly used. Seems weird to have to cast the same var 5 different
>> ways. Definitely agree that one would likely deal with this by
>> wrapping, but then you end up either needing to extend your wrapper
>> or add new wrappers...
> Okay, I see, all of that is going to happen in one Consumer; your not
> going to want to read the TokenStream more than once. I see your point
> now.
>
Hmm, I'm not sure I understand why you would have to read the
TokenStream more than once?
-Michael
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Mark Miller <ma...@gmail.com>.
Grant Ingersoll wrote:
>
> On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
>
>> Grant Ingersoll wrote:
>>>
>>> Bear with me, b/c I'm not sure I'm following, but looking at
>>> https://issues.apache.org/jira/browse/LUCENE-1422, I see at least 5
>>> different implemented Attributes.
>>>
>>> So, let's say I add a 5 more attributes and now have a total of 10
>>> attributes. Are you saying that I then would have, potentially, 10
>>> different variables that all point to the token as in the code
>>> snippet above where the casting takes place? Or would I just create
>>> a single "Super" attribute that folds in all of my new attributes,
>>> plus any other existing ones? Or, maybe, what I would do is create
>>> the 5 new attributes and then 1 new attribute that extends all 10,
>>> thus allowing me to use them individually, but saving me from having
>>> to do a whole ton of casting in my Consumer.
>> Potentially one consumer doing 10 things, but not likely right? I
>> mean, things will stay logical as they are now, and rather than a
>> super consumer doing everything, we will still have a chain of
>> consumers each doing its own piece. So more likely, maybe something
>> comes along every so often (another 5, over *much* time, say) and
>> each time we add a Consumer that uses one or two TokenStream types.
>> And then its just an implementation detail on whether you make a
>> composite TokenStream - if you have added 10 new attributes and see
>> it fit to make one consumer use them all, sure, make a composite,
>> super type, but in my mind, the way its done in the example code is
>> clearer/cleaner for a handful of TokenStream types. And even if you
>> do make the composite,super type, its likely to just be a sugar
>> wrapper anyway - the implementation for say, payload and positions,
>> should probably be maintained in their own classes anyway.
>
> Well, there are 5 different attributes already, all of which are
> commonly used. Seems weird to have to cast the same var 5 different
> ways. Definitely agree that one would likely deal with this by
> wrapping, but then you end up either needing to extend your wrapper or
> add new wrappers...
Okay, I see, all of that is going to happen in one Consumer; your not
going to want to read the TokenStream more than once. I see your point now.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Michael Busch <bu...@gmail.com>.
Grant Ingersoll wrote:
>
> On Oct 21, 2008, at 1:39 AM, Michael Busch wrote:
>>>
>>> Perhaps it would be useful for Lucene to offer exactly one subclass
>>> of Token that we guarantee will always have all known Attributes
>>> (i.e. the ones Lucene provides) available to it for casting purposes.
>>>
>>
>> Yeah we could do that. In fact, I did exactly this when I started
>> working on this patch. I created a class called PlainToken, which had
>> all the termBuffer and attributes logic, and changed Token to extend
>> it. Then the new getToken() method would return an instance of
>> PlainToken. My main concern with this approach is that it will make
>> the code in the indexer more complicated, because it always has to
>> check if we have a Token or PlainToken; if it's a Token then it has
>> to use the get*() method directly, for a PlainToken it has tocheck
>> for the *Attributes. So that's a bit messy (it's in fact exactly like
>> that in the current patch for backwards-compatibility, but we could
>> clean this up in 3.0). So for code simplicity I'm slightly in favor
>> of not creating the a class that implements a default set of
>> functionality without Attributes.
>
> Yes that would be messy, but not exactly what I was proposing. I was
> originally thinking we needed a derived class, but now it seems like
> we should just keep convenience methods on Token itself.
>
> That is, why not just have Token implement both the attribute methods
> and dummy wrappers for the guaranteed to exist Attributes that Lucene
> implements?
>
> e.g.
>
> public int startOffset(){
> return getAttribute(OffsetAttribute.class).endOffset();
> }
>
> This makes back-compat a snap, moreover it causes less pain for
> people, b/c Analyzer/Token stuff is more than likely the one of the
> most customized pieces of Lucene.
>
That's a good idea! I will add that to my patch...
-Michael
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 21, 2008, at 1:39 AM, Michael Busch wrote:
>>
>> Perhaps it would be useful for Lucene to offer exactly one subclass
>> of Token that we guarantee will always have all known Attributes
>> (i.e. the ones Lucene provides) available to it for casting
>> purposes.
>>
>
> Yeah we could do that. In fact, I did exactly this when I started
> working on this patch. I created a class called PlainToken, which
> had all the termBuffer and attributes logic, and changed Token to
> extend it. Then the new getToken() method would return an instance
> of PlainToken. My main concern with this approach is that it will
> make the code in the indexer more complicated, because it always has
> to check if we have a Token or PlainToken; if it's a Token then it
> has to use the get*() method directly, for a PlainToken it has
> tocheck for the *Attributes. So that's a bit messy (it's in fact
> exactly like that in the current patch for backwards-compatibility,
> but we could clean this up in 3.0). So for code simplicity I'm
> slightly in favor of not creating the a class that implements a
> default set of functionality without Attributes.
Yes that would be messy, but not exactly what I was proposing. I was
originally thinking we needed a derived class, but now it seems like
we should just keep convenience methods on Token itself.
That is, why not just have Token implement both the attribute methods
and dummy wrappers for the guaranteed to exist Attributes that Lucene
implements?
e.g.
public int startOffset(){
return getAttribute(OffsetAttribute.class).endOffset();
}
This makes back-compat a snap, moreover it causes less pain for
people, b/c Analyzer/Token stuff is more than likely the one of the
most customized pieces of Lucene.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Michael Busch <bu...@gmail.com>.
Grant Ingersoll wrote:
>
> On Oct 19, 2008, at 7:08 PM, Michael Busch wrote:
>
>> Grant Ingersoll wrote:
>>>
>>> On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
>>>
>>>> Grant Ingersoll wrote:
>>>>>
>>>>> Bear with me, b/c I'm not sure I'm following, but looking at
>>>>> https://issues.apache.org/jira/browse/LUCENE-1422, I see at least
>>>>> 5 different implemented Attributes.
>>>>>
>>>>> So, let's say I add a 5 more attributes and now have a total of 10
>>>>> attributes. Are you saying that I then would have, potentially, 10
>>>>> different variables that all point to the token as in the code
>>>>> snippet above where the casting takes place? Or would I just
>>>>> create a single "Super" attribute that folds in all of my new
>>>>> attributes, plus any other existing ones? Or, maybe, what I would
>>>>> do is create the 5 new attributes and then 1 new attribute that
>>>>> extends all 10, thus allowing me to use them individually, but
>>>>> saving me from having to do a whole ton of casting in my Consumer.
>>>> Potentially one consumer doing 10 things, but not likely right? I
>>>> mean, things will stay logical as they are now, and rather than a
>>>> super consumer doing everything, we will still have a chain of
>>>> consumers each doing its own piece. So more likely, maybe something
>>>> comes along every so often (another 5, over *much* time, say) and
>>>> each time we add a Consumer that uses one or two TokenStream types.
>>>> And then its just an implementation detail on whether you make a
>>>> composite TokenStream - if you have added 10 new attributes and see
>>>> it fit to make one consumer use them all, sure, make a composite,
>>>> super type, but in my mind, the way its done in the example code is
>>>> clearer/cleaner for a handful of TokenStream types. And even if you
>>>> do make the composite,super type, its likely to just be a sugar
>>>> wrapper anyway - the implementation for say, payload and positions,
>>>> should probably be maintained in their own classes anyway.
>>>
>>> Well, there are 5 different attributes already, all of which are
>>> commonly used. Seems weird to have to cast the same var 5 different
>>> ways. Definitely agree that one would likely deal with this by
>>> wrapping, but then you end up either needing to extend your wrapper
>>> or add new wrappers...
>>
>> Well yes, there are 5 attributes, but n neither of the core
>> tokenstreams and -filters that I changed in my patch did I have to
>> use more than two or three of those. Currently the only attributes
>> that are really used are PositionIncrementAttribute and
>> PayloadAttribute. And the OffsetAttribute when TermVectors are turned
>> on.
>>
>> Even in the indexing chain currently we don't have a single consumer
>> that needs all attributes. The FreqProxWriter needs positions and
>> payloads, the TermVectorsWriter needs positions and offsets.
>
>
> I have an application that uses all the attributes of a Token, or at
> least, almost all of them. There are many uses for Lucene's analysis
> code that have nothing to do with indexing, Consumers or even Lucene.
>
>>
>>
>> Also, you don't have to cast the same variable multiple times. In the
>> current patch you would call e. g.
>> token.getAttribute(PayloadAttribute.class) and keep a reference to it
>> in the consumer or filter.
>>
>> IMO even calling getAttribute() 5 times or so and storing the
>> references wouldn't be so bad. And if you really don't like it you
>> could make a wrapper as you said. You also mentioned the
>> disadvantages of the wrapper, e. g. that you would have to extend it
>> to add new attributes. But then, isn't that the same disadvantage the
>> current Token API has?
>
> True. I didn't say the idea was bad, in fact I mostly like it, I was
> just saying I'd like to explore how it would work in practice and the
> main thing that struck me was all the casting or all the references.
> Since it's likely that you only deal with a Token one at a time,
> you're right, it's probably not a big deal other than the code looks
> funny, IMO.
>
>>
>>
>> You could even use the new API in exact the same way as the old one.
>> Just create a subclass of Token that has all members you need and
>> don't add any attributes.
>>
>> So I think the new API adds more flexibility, and still offers to use
>> it in the same way as the old one. I however think the recommended
>> best practice should be to use the new attributes, for reusability of
>> consumers that only need certain attributes.
>
> Perhaps it would be useful for Lucene to offer exactly one subclass of
> Token that we guarantee will always have all known Attributes (i.e.
> the ones Lucene provides) available to it for casting purposes.
>
Yeah we could do that. In fact, I did exactly this when I started
working on this patch. I created a class called PlainToken, which had
all the termBuffer and attributes logic, and changed Token to extend it.
Then the new getToken() method would return an instance of PlainToken.
My main concern with this approach is that it will make the code in the
indexer more complicated, because it always has to check if we have a
Token or PlainToken; if it's a Token then it has to use the get*()
method directly, for a PlainToken it has tocheck for the *Attributes. So
that's a bit messy (it's in fact exactly like that in the current patch
for backwards-compatibility, but we could clean this up in 3.0). So for
code simplicity I'm slightly in favor of not creating the a class that
implements a default set of functionality without Attributes.
-Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 19, 2008, at 7:08 PM, Michael Busch wrote:
> Grant Ingersoll wrote:
>>
>> On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
>>
>>> Grant Ingersoll wrote:
>>>>
>>>> Bear with me, b/c I'm not sure I'm following, but looking at https://issues.apache.org/jira/browse/LUCENE-1422
>>>> , I see at least 5 different implemented Attributes.
>>>>
>>>> So, let's say I add a 5 more attributes and now have a total of
>>>> 10 attributes. Are you saying that I then would have,
>>>> potentially, 10 different variables that all point to the token
>>>> as in the code snippet above where the casting takes place? Or
>>>> would I just create a single "Super" attribute that folds in all
>>>> of my new attributes, plus any other existing ones? Or, maybe,
>>>> what I would do is create the 5 new attributes and then 1 new
>>>> attribute that extends all 10, thus allowing me to use them
>>>> individually, but saving me from having to do a whole ton of
>>>> casting in my Consumer.
>>> Potentially one consumer doing 10 things, but not likely right? I
>>> mean, things will stay logical as they are now, and rather than a
>>> super consumer doing everything, we will still have a chain of
>>> consumers each doing its own piece. So more likely, maybe
>>> something comes along every so often (another 5, over *much* time,
>>> say) and each time we add a Consumer that uses one or two
>>> TokenStream types. And then its just an implementation detail on
>>> whether you make a composite TokenStream - if you have added 10
>>> new attributes and see it fit to make one consumer use them all,
>>> sure, make a composite, super type, but in my mind, the way its
>>> done in the example code is clearer/cleaner for a handful of
>>> TokenStream types. And even if you do make the composite,super
>>> type, its likely to just be a sugar wrapper anyway - the
>>> implementation for say, payload and positions, should probably be
>>> maintained in their own classes anyway.
>>
>> Well, there are 5 different attributes already, all of which are
>> commonly used. Seems weird to have to cast the same var 5
>> different ways. Definitely agree that one would likely deal with
>> this by wrapping, but then you end up either needing to extend your
>> wrapper or add new wrappers...
>
> Well yes, there are 5 attributes, but n neither of the core
> tokenstreams and -filters that I changed in my patch did I have to
> use more than two or three of those. Currently the only attributes
> that are really used are PositionIncrementAttribute and
> PayloadAttribute. And the OffsetAttribute when TermVectors are
> turned on.
>
> Even in the indexing chain currently we don't have a single consumer
> that needs all attributes. The FreqProxWriter needs positions and
> payloads, the TermVectorsWriter needs positions and offsets.
I have an application that uses all the attributes of a Token, or at
least, almost all of them. There are many uses for Lucene's analysis
code that have nothing to do with indexing, Consumers or even Lucene.
>
>
> Also, you don't have to cast the same variable multiple times. In
> the current patch you would call e. g.
> token.getAttribute(PayloadAttribute.class) and keep a reference to
> it in the consumer or filter.
>
> IMO even calling getAttribute() 5 times or so and storing the
> references wouldn't be so bad. And if you really don't like it you
> could make a wrapper as you said. You also mentioned the
> disadvantages of the wrapper, e. g. that you would have to extend it
> to add new attributes. But then, isn't that the same disadvantage
> the current Token API has?
True. I didn't say the idea was bad, in fact I mostly like it, I was
just saying I'd like to explore how it would work in practice and the
main thing that struck me was all the casting or all the references.
Since it's likely that you only deal with a Token one at a time,
you're right, it's probably not a big deal other than the code looks
funny, IMO.
>
>
> You could even use the new API in exact the same way as the old one.
> Just create a subclass of Token that has all members you need and
> don't add any attributes.
>
> So I think the new API adds more flexibility, and still offers to
> use it in the same way as the old one. I however think the
> recommended best practice should be to use the new attributes, for
> reusability of consumers that only need certain attributes.
Perhaps it would be useful for Lucene to offer exactly one subclass of
Token that we guarantee will always have all known Attributes (i.e.
the ones Lucene provides) available to it for casting purposes.
>
>
> However, please let me know if you have any concrete recommendations
> about changing the API in LUCENE-1422.
I thought those concerns were pretty concrete... :-)
> There might be better ones than the APIs I came up with.
I think the APIs in the 2nd patch look pretty reasonable.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Michael Busch <bu...@gmail.com>.
Grant Ingersoll wrote:
>
> On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
>
>> Grant Ingersoll wrote:
>>>
>>> Bear with me, b/c I'm not sure I'm following, but looking at
>>> https://issues.apache.org/jira/browse/LUCENE-1422, I see at least 5
>>> different implemented Attributes.
>>>
>>> So, let's say I add a 5 more attributes and now have a total of 10
>>> attributes. Are you saying that I then would have, potentially, 10
>>> different variables that all point to the token as in the code
>>> snippet above where the casting takes place? Or would I just create
>>> a single "Super" attribute that folds in all of my new attributes,
>>> plus any other existing ones? Or, maybe, what I would do is create
>>> the 5 new attributes and then 1 new attribute that extends all 10,
>>> thus allowing me to use them individually, but saving me from having
>>> to do a whole ton of casting in my Consumer.
>> Potentially one consumer doing 10 things, but not likely right? I
>> mean, things will stay logical as they are now, and rather than a
>> super consumer doing everything, we will still have a chain of
>> consumers each doing its own piece. So more likely, maybe something
>> comes along every so often (another 5, over *much* time, say) and
>> each time we add a Consumer that uses one or two TokenStream types.
>> And then its just an implementation detail on whether you make a
>> composite TokenStream - if you have added 10 new attributes and see
>> it fit to make one consumer use them all, sure, make a composite,
>> super type, but in my mind, the way its done in the example code is
>> clearer/cleaner for a handful of TokenStream types. And even if you
>> do make the composite,super type, its likely to just be a sugar
>> wrapper anyway - the implementation for say, payload and positions,
>> should probably be maintained in their own classes anyway.
>
> Well, there are 5 different attributes already, all of which are
> commonly used. Seems weird to have to cast the same var 5 different
> ways. Definitely agree that one would likely deal with this by
> wrapping, but then you end up either needing to extend your wrapper or
> add new wrappers...
Well yes, there are 5 attributes, but n neither of the core tokenstreams
and -filters that I changed in my patch did I have to use more than two
or three of those. Currently the only attributes that are really used
are PositionIncrementAttribute and PayloadAttribute. And the
OffsetAttribute when TermVectors are turned on.
Even in the indexing chain currently we don't have a single consumer
that needs all attributes. The FreqProxWriter needs positions and
payloads, the TermVectorsWriter needs positions and offsets.
Also, you don't have to cast the same variable multiple times. In the
current patch you would call e. g.
token.getAttribute(PayloadAttribute.class) and keep a reference to it in
the consumer or filter.
IMO even calling getAttribute() 5 times or so and storing the references
wouldn't be so bad. And if you really don't like it you could make a
wrapper as you said. You also mentioned the disadvantages of the
wrapper, e. g. that you would have to extend it to add new attributes.
But then, isn't that the same disadvantage the current Token API has?
You could even use the new API in exact the same way as the old one.
Just create a subclass of Token that has all members you need and don't
add any attributes.
So I think the new API adds more flexibility, and still offers to use it
in the same way as the old one. I however think the recommended best
practice should be to use the new attributes, for reusability of
consumers that only need certain attributes.
However, please let me know if you have any concrete recommendations
about changing the API in LUCENE-1422. There might be better ones than
the APIs I came up with. It would be good to attach some code & demo
usage, then it's much easier to talk about it and see the
(dis)advantages of the different approaches.
-Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 19, 2008, at 12:56 AM, Mark Miller wrote:
> Grant Ingersoll wrote:
>>
>> Bear with me, b/c I'm not sure I'm following, but looking at https://issues.apache.org/jira/browse/LUCENE-1422
>> , I see at least 5 different implemented Attributes.
>>
>> So, let's say I add a 5 more attributes and now have a total of 10
>> attributes. Are you saying that I then would have, potentially, 10
>> different variables that all point to the token as in the code
>> snippet above where the casting takes place? Or would I just create
>> a single "Super" attribute that folds in all of my new attributes,
>> plus any other existing ones? Or, maybe, what I would do is create
>> the 5 new attributes and then 1 new attribute that extends all 10,
>> thus allowing me to use them individually, but saving me from
>> having to do a whole ton of casting in my Consumer.
> Potentially one consumer doing 10 things, but not likely right? I
> mean, things will stay logical as they are now, and rather than a
> super consumer doing everything, we will still have a chain of
> consumers each doing its own piece. So more likely, maybe something
> comes along every so often (another 5, over *much* time, say) and
> each time we add a Consumer that uses one or two TokenStream types.
> And then its just an implementation detail on whether you make a
> composite TokenStream - if you have added 10 new attributes and see
> it fit to make one consumer use them all, sure, make a composite,
> super type, but in my mind, the way its done in the example code is
> clearer/cleaner for a handful of TokenStream types. And even if you
> do make the composite,super type, its likely to just be a sugar
> wrapper anyway - the implementation for say, payload and positions,
> should probably be maintained in their own classes anyway.
Well, there are 5 different attributes already, all of which are
commonly used. Seems weird to have to cast the same var 5 different
ways. Definitely agree that one would likely deal with this by
wrapping, but then you end up either needing to extend your wrapper or
add new wrappers...
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Mark Miller <ma...@gmail.com>.
Grant Ingersoll wrote:
>
> Bear with me, b/c I'm not sure I'm following, but looking at
> https://issues.apache.org/jira/browse/LUCENE-1422, I see at least 5
> different implemented Attributes.
>
> So, let's say I add a 5 more attributes and now have a total of 10
> attributes. Are you saying that I then would have, potentially, 10
> different variables that all point to the token as in the code snippet
> above where the casting takes place? Or would I just create a single
> "Super" attribute that folds in all of my new attributes, plus any
> other existing ones? Or, maybe, what I would do is create the 5 new
> attributes and then 1 new attribute that extends all 10, thus allowing
> me to use them individually, but saving me from having to do a whole
> ton of casting in my Consumer.
Potentially one consumer doing 10 things, but not likely right? I mean,
things will stay logical as they are now, and rather than a super
consumer doing everything, we will still have a chain of consumers each
doing its own piece. So more likely, maybe something comes along every
so often (another 5, over *much* time, say) and each time we add a
Consumer that uses one or two TokenStream types. And then its just an
implementation detail on whether you make a composite TokenStream - if
you have added 10 new attributes and see it fit to make one consumer use
them all, sure, make a composite, super type, but in my mind, the way
its done in the example code is clearer/cleaner for a handful of
TokenStream types. And even if you do make the composite,super type, its
likely to just be a sugar wrapper anyway - the implementation for say,
payload and positions, should probably be maintained in their own
classes anyway.
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 11, 2008, at 7:33 PM, Michael Busch wrote:
> If a consumer is e. g. only interested in the positionIncrement and
> Payload, it can consume the tokens like this:
>
> public class Consumer {
> public void consumeTokens(TokenStream ts) throws IOException {
> Token token = ts.prototypeToken();
>
> PayloadAttribute payloadSource = (PayloadAttribute) token;
> PositionIncrementAttribute positionSource =
> (PositionIncrementAttribute) token;
>
> while (ts.nextToken()) {
> char[] term = token.termBuffer();
> int termLength = token.termLength();
> int positionIncrement = positionSource.getPositionIncrement();
> Payload payload = payloadSource.getPayload();
>
> // do something with the term, positionIncrement and payload
> }
> }
> }
>
Bear with me, b/c I'm not sure I'm following, but looking at https://issues.apache.org/jira/browse/LUCENE-1422
, I see at least 5 different implemented Attributes.
So, let's say I add a 5 more attributes and now have a total of 10
attributes. Are you saying that I then would have, potentially, 10
different variables that all point to the token as in the code snippet
above where the casting takes place? Or would I just create a single
"Super" attribute that folds in all of my new attributes, plus any
other existing ones? Or, maybe, what I would do is create the 5 new
attributes and then 1 new attribute that extends all 10, thus allowing
me to use them individually, but saving me from having to do a whole
ton of casting in my Consumer.
I like where this is going, but am still a bit fuzzy on how it's going
to work in practice.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Shai Erera <se...@gmail.com>.
Thanks Michael
LUCENE-1301 was exactly what I was looking for to complete my understanding.
All is clear now
On Sun, Oct 12, 2008 at 10:42 AM, Michael Busch <bu...@gmail.com> wrote:
> Hi Shai,
>
> I'm going to shuffle your email a bit to answer in a different order...
>
> Shai Erera wrote:
> > Perhaps what you write below is linked to another thread (on flexible
> > indexing maybe?) which I'm not aware of, so I'd appreciate if you can
> > give me a reference.
>
> Mike recently refactored the DocumentsWriter into a bunch of new classes
> (see LUCENE-1301). In fact, we now have an indexer chain, in which you can
> plugin different modules that do something (well, we still have to add an
> API to make use of the chain...)
>
> For example, there are currently two TermsHashConsumers in the default
> chain: the FreqProxTermsWriter and the TermVectorTermsWriter. Both consume
> the tokens from a TokenStream and write the different data structures.
>
> We could for example write a SpanPostingTermsHashConsumer that can not only
> write the start position of a token but also the number of covered
> positions. We could introduce a new interface:
> public interface SpanAttribute extends PositionIncrementAttribute {
> public int getLength();
> public int setLength(int length);
> }
>
> Only the SpanPostingTermsHashConsumer would need to know the SpanAttribute.
>
>
>> BTW, what I didn't understand from you description is how does the
>> indexing part know which attributes my Token supports? For example, let's
>> say I create a Token which implements only position increments, no payload
>> and perhaps some other custom attribute. I generate a TokenStream returning
>> this Token type.
>> How will Lucene's indexing mechanism know my Token supports only position
>> increments and especially the custom attribute? What will it do with that
>> custom attribute?
>>
>
> The advantage is that the different consumers actually don't need to know
> the exact type of the Token. Each consumer can check via instanceof if the
> prototype Token actually implements the interface(s) that the consumer
> needs. If not, then the consumer can just not process the tokens for that
> particular field. Alternatively we could say that the user needs to make
> sure that the appropriate prototype Token is generated for the indexing
> chain that is configured, otherwise Lucene throws an Exception.
>
> I think the main advantage here is that we can implement consumers that
> only care about particular attributes. Btw, Doug had actually a very similar
> idea for the Token class that he mentioned almost 2 years ago:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43486#43486
>
> > In 3.0 you plan to move to Java 1.5, right? Couldn't you use the Java
> > templates then? Have the calling application pass in the Token
> > template it wants to use and then the consumer does not need to cast
> > anything ...
>
> That only works if we keep the current design in which the consumer has to
> create the Token. But what do you do if you have more than one consumer? (E.
> g. adding a new TermsHashConsumer into the chain?)
>
> -Michael
>
>
>> Shai
>>
>>
>> On Sun, Oct 12, 2008 at 1:33 AM, Michael Busch <buschmic@gmail.com<mailto:
>> buschmic@gmail.com>> wrote:
>>
>> Hi,
>>
>> I've been thinking about making the TokenStream and Token APIs more
>> flexible. E. g. for fields that don't store positions, the Token
>> doesn't need to have a positionIncrement or a payload. With flexible
>> indexing on the other hand, people might want to add custom
>> attributes to a Token that a consumer in the indexing chain could
>> use then.
>>
>> Of course it is possible to extend Token, because it is not final,
>> and add additional attributes to it. But then consumers of the
>> TokenStream must downcast every instance of the Token object when
>> they call next(Token).
>>
>> I was therefore thinking about a different TokenStream API:
>>
>> public abstract class TokenStream {
>> public abstract boolean nextToken() throws IOException;
>>
>> public abstract Token prototypeToken() throws IOException;
>>
>> public void reset() throws IOException {}
>>
>> public void close() throws IOException {}
>> }
>>
>> Furthermore Token itself would only keep the termBuffer logic and we
>> could introduce different interfaces, like:
>>
>> public interface PayloadAttribute {
>> /**
>> * Returns this Token's payload.
>> */
>> public Payload getPayload();
>>
>> /**
>> * Sets this Token's payload.
>> */
>> public void setPayload(Payload payload);
>> }
>>
>> public interface PositionIncrementAttribute {
>> /** Set the position increment. This determines the position of
>> * this token relative to the previous Token in a
>> * {@link TokenStream}, used in phrase searching.
>> */
>> public void setPositionIncrement(int positionIncrement);
>>
>> /** Returns the position increment of this Token.
>> * @see #setPositionIncrement
>> */
>> public int getPositionIncrement();
>> }
>>
>> A consumer, e. g. the DocumentsWriter, does not create a Token
>> instance itself anymore, but rather calls prototypeToken(). This
>> method returns a Token subclass which implements all desired
>> *Attribute interfaces.
>>
>> If a consumer is e. g. only interested in the positionIncrement and
>> Payload, it can consume the tokens like this:
>>
>> public class Consumer {
>> public void consumeTokens(TokenStream ts) throws IOException {
>> Token token = ts.prototypeToken();
>>
>> PayloadAttribute payloadSource = (PayloadAttribute) token;
>> PositionIncrementAttribute positionSource =
>> (PositionIncrementAttribute) token;
>>
>> while (ts.nextToken()) {
>> char[] term = token.termBuffer();
>> int termLength = token.termLength();
>> int positionIncrement = positionSource.getPositionIncrement();
>> Payload payload = payloadSource.getPayload();
>>
>> // do something with the term, positionIncrement and payload
>> }
>> }
>> }
>>
>> Casting is now only done once after the prototype token was created.
>> Now if you want to add another consumer in the indexing chain and
>> realize that you want to add another attribute to the Token, then
>> you don't have to change this consumer. You only need to create
>> another Token subclass that implements the new attribute in addition
>> to the previous ones and can use it in the new consumer.
>>
>> I haven't tried to implement this yet and maybe there are things I
>> haven't thought about (like caching TokenFilters). I'd like to get
>> some feedback about these APIs first to see if this makes sense?
>>
>> Btw: if we think this (or another) approach to change these APIs
>> makes sense, then it would be good to change it for 3.0 when we can
>> break backwards compatibility. And then we should also rethink the
>> Fieldable/AbstractField/Field/FieldInfos APIs for 3.0 and flexible
>> indexing!
>>
>> -Michael
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> <ma...@lucene.apache.org>
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>> <ma...@lucene.apache.org>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: TokenStream and Token APIs
Posted by Michael Busch <bu...@gmail.com>.
Hi Shai,
I'm going to shuffle your email a bit to answer in a different order...
Shai Erera wrote:
> Perhaps what you write below is linked to another thread (on flexible
> indexing maybe?) which I'm not aware of, so I'd appreciate if you can
> give me a reference.
Mike recently refactored the DocumentsWriter into a bunch of new classes
(see LUCENE-1301). In fact, we now have an indexer chain, in which you
can plugin different modules that do something (well, we still have to
add an API to make use of the chain...)
For example, there are currently two TermsHashConsumers in the default
chain: the FreqProxTermsWriter and the TermVectorTermsWriter. Both
consume the tokens from a TokenStream and write the different data
structures.
We could for example write a SpanPostingTermsHashConsumer that can not
only write the start position of a token but also the number of covered
positions. We could introduce a new interface:
public interface SpanAttribute extends PositionIncrementAttribute {
public int getLength();
public int setLength(int length);
}
Only the SpanPostingTermsHashConsumer would need to know the SpanAttribute.
>
> BTW, what I didn't understand from you description is how does the
> indexing part know which attributes my Token supports? For example,
> let's say I create a Token which implements only position increments, no
> payload and perhaps some other custom attribute. I generate a
> TokenStream returning this Token type.
> How will Lucene's indexing mechanism know my Token supports only
> position increments and especially the custom attribute? What will it do
> with that custom attribute?
The advantage is that the different consumers actually don't need to
know the exact type of the Token. Each consumer can check via instanceof
if the prototype Token actually implements the interface(s) that the
consumer needs. If not, then the consumer can just not process the
tokens for that particular field. Alternatively we could say that the
user needs to make sure that the appropriate prototype Token is
generated for the indexing chain that is configured, otherwise Lucene
throws an Exception.
I think the main advantage here is that we can implement consumers that
only care about particular attributes. Btw, Doug had actually a very
similar idea for the Token class that he mentioned almost 2 years ago:
http://www.gossamer-threads.com/lists/lucene/java-dev/43486#43486
> In 3.0 you plan to move to Java 1.5, right? Couldn't you use the Java
> templates then? Have the calling application pass in the Token
> template it wants to use and then the consumer does not need to cast
> anything ...
That only works if we keep the current design in which the consumer has
to create the Token. But what do you do if you have more than one
consumer? (E. g. adding a new TermsHashConsumer into the chain?)
-Michael
>
> Shai
>
> On Sun, Oct 12, 2008 at 1:33 AM, Michael Busch <buschmic@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> I've been thinking about making the TokenStream and Token APIs more
> flexible. E. g. for fields that don't store positions, the Token
> doesn't need to have a positionIncrement or a payload. With flexible
> indexing on the other hand, people might want to add custom
> attributes to a Token that a consumer in the indexing chain could
> use then.
>
> Of course it is possible to extend Token, because it is not final,
> and add additional attributes to it. But then consumers of the
> TokenStream must downcast every instance of the Token object when
> they call next(Token).
>
> I was therefore thinking about a different TokenStream API:
>
> public abstract class TokenStream {
> public abstract boolean nextToken() throws IOException;
>
> public abstract Token prototypeToken() throws IOException;
>
> public void reset() throws IOException {}
>
> public void close() throws IOException {}
> }
>
> Furthermore Token itself would only keep the termBuffer logic and we
> could introduce different interfaces, like:
>
> public interface PayloadAttribute {
> /**
> * Returns this Token's payload.
> */
> public Payload getPayload();
>
> /**
> * Sets this Token's payload.
> */
> public void setPayload(Payload payload);
> }
>
> public interface PositionIncrementAttribute {
> /** Set the position increment. This determines the position of
> * this token relative to the previous Token in a
> * {@link TokenStream}, used in phrase searching.
> */
> public void setPositionIncrement(int positionIncrement);
>
> /** Returns the position increment of this Token.
> * @see #setPositionIncrement
> */
> public int getPositionIncrement();
> }
>
> A consumer, e. g. the DocumentsWriter, does not create a Token
> instance itself anymore, but rather calls prototypeToken(). This
> method returns a Token subclass which implements all desired
> *Attribute interfaces.
>
> If a consumer is e. g. only interested in the positionIncrement and
> Payload, it can consume the tokens like this:
>
> public class Consumer {
> public void consumeTokens(TokenStream ts) throws IOException {
> Token token = ts.prototypeToken();
>
> PayloadAttribute payloadSource = (PayloadAttribute) token;
> PositionIncrementAttribute positionSource =
> (PositionIncrementAttribute) token;
>
> while (ts.nextToken()) {
> char[] term = token.termBuffer();
> int termLength = token.termLength();
> int positionIncrement = positionSource.getPositionIncrement();
> Payload payload = payloadSource.getPayload();
>
> // do something with the term, positionIncrement and payload
> }
> }
> }
>
> Casting is now only done once after the prototype token was created.
> Now if you want to add another consumer in the indexing chain and
> realize that you want to add another attribute to the Token, then
> you don't have to change this consumer. You only need to create
> another Token subclass that implements the new attribute in addition
> to the previous ones and can use it in the new consumer.
>
> I haven't tried to implement this yet and maybe there are things I
> haven't thought about (like caching TokenFilters). I'd like to get
> some feedback about these APIs first to see if this makes sense?
>
> Btw: if we think this (or another) approach to change these APIs
> makes sense, then it would be good to change it for 3.0 when we can
> break backwards compatibility. And then we should also rethink the
> Fieldable/AbstractField/Field/FieldInfos APIs for 3.0 and flexible
> indexing!
>
> -Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> <ma...@lucene.apache.org>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Shai Erera <se...@gmail.com>.
In 3.0 you plan to move to Java 1.5, right? Couldn't you use the Java
templates then? Have the calling application pass in the Token template it
wants to use and then the consumer does not need to cast anything ...
BTW, what I didn't understand from you description is how does the indexing
part know which attributes my Token supports? For example, let's say I
create a Token which implements only position increments, no payload and
perhaps some other custom attribute. I generate a TokenStream returning this
Token type.
How will Lucene's indexing mechanism know my Token supports only position
increments and especially the custom attribute? What will it do with that
custom attribute?
Perhaps what you write below is linked to another thread (on flexible
indexing maybe?) which I'm not aware of, so I'd appreciate if you can give
me a reference.
Shai
On Sun, Oct 12, 2008 at 1:33 AM, Michael Busch <bu...@gmail.com> wrote:
> Hi,
>
> I've been thinking about making the TokenStream and Token APIs more
> flexible. E. g. for fields that don't store positions, the Token doesn't
> need to have a positionIncrement or a payload. With flexible indexing on the
> other hand, people might want to add custom attributes to a Token that a
> consumer in the indexing chain could use then.
>
> Of course it is possible to extend Token, because it is not final, and add
> additional attributes to it. But then consumers of the TokenStream must
> downcast every instance of the Token object when they call next(Token).
>
> I was therefore thinking about a different TokenStream API:
>
> public abstract class TokenStream {
> public abstract boolean nextToken() throws IOException;
>
> public abstract Token prototypeToken() throws IOException;
>
> public void reset() throws IOException {}
>
> public void close() throws IOException {}
> }
>
> Furthermore Token itself would only keep the termBuffer logic and we could
> introduce different interfaces, like:
>
> public interface PayloadAttribute {
> /**
> * Returns this Token's payload.
> */
> public Payload getPayload();
>
> /**
> * Sets this Token's payload.
> */
> public void setPayload(Payload payload);
> }
>
> public interface PositionIncrementAttribute {
> /** Set the position increment. This determines the position of
> * this token relative to the previous Token in a
> * {@link TokenStream}, used in phrase searching.
> */
> public void setPositionIncrement(int positionIncrement);
>
> /** Returns the position increment of this Token.
> * @see #setPositionIncrement
> */
> public int getPositionIncrement();
> }
>
> A consumer, e. g. the DocumentsWriter, does not create a Token instance
> itself anymore, but rather calls prototypeToken(). This method returns a
> Token subclass which implements all desired *Attribute interfaces.
>
> If a consumer is e. g. only interested in the positionIncrement and
> Payload, it can consume the tokens like this:
>
> public class Consumer {
> public void consumeTokens(TokenStream ts) throws IOException {
> Token token = ts.prototypeToken();
>
> PayloadAttribute payloadSource = (PayloadAttribute) token;
> PositionIncrementAttribute positionSource =
> (PositionIncrementAttribute) token;
>
> while (ts.nextToken()) {
> char[] term = token.termBuffer();
> int termLength = token.termLength();
> int positionIncrement = positionSource.getPositionIncrement();
> Payload payload = payloadSource.getPayload();
>
> // do something with the term, positionIncrement and payload
> }
> }
> }
>
> Casting is now only done once after the prototype token was created. Now if
> you want to add another consumer in the indexing chain and realize that you
> want to add another attribute to the Token, then you don't have to change
> this consumer. You only need to create another Token subclass that
> implements the new attribute in addition to the previous ones and can use it
> in the new consumer.
>
> I haven't tried to implement this yet and maybe there are things I haven't
> thought about (like caching TokenFilters). I'd like to get some feedback
> about these APIs first to see if this makes sense?
>
> Btw: if we think this (or another) approach to change these APIs makes
> sense, then it would be good to change it for 3.0 when we can break
> backwards compatibility. And then we should also rethink the
> Fieldable/AbstractField/Field/FieldInfos APIs for 3.0 and flexible indexing!
>
> -Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: TokenStream and Token APIs
Posted by DM Smith <dm...@gmail.com>.
On Oct 13, 2008, at 3:34 PM, Doug Cutting wrote:
> Michael Busch wrote:
>> public abstract boolean nextToken() throws IOException;
>
> What's the point of a separate Token and TokenStream if there's only
> a single Token per TokenStream? If that's really the direction
> we'll go, then all of the Token methods should be on TokenStream,
> and Token should disappear. Are there cases where a stream might
> switch token classes midstream? If not, then a single, combined API
> should suffice.
There are several streams that analyze the input and output several
tokens for each one in the stream. For example, synonyms, shingles,
stemming.
There are also some caching TokenStreams that can be reset to replay
their stream.
-- DM
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: TokenStream and Token APIs
Posted by Doug Cutting <cu...@apache.org>.
Michael Busch wrote:
> public abstract boolean nextToken() throws IOException;
What's the point of a separate Token and TokenStream if there's only a
single Token per TokenStream? If that's really the direction we'll go,
then all of the Token methods should be on TokenStream, and Token should
disappear. Are there cases where a stream might switch token classes
midstream? If not, then a single, combined API should suffice.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org