You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Aad Nales <aa...@gmail.com> on 2010/06/07 16:43:20 UTC
How to add "tokens" to a stream
Hi All,
Years ago we implemented a Lucene solution which we are updating
today, and i am a bit lost on the following.
In Lucene 1.x and 2.x it was possible to add a token in a Filter
simply by returning an extra Token when next was being called. What i
can not find is an equivalent possiblity for adding a Token in a
filter. The incrementToken() seem to be the logical place to implement
similar functionality (at least it could trigger it) but i am a bit
lost how to access the stream and insert an extra token. Any
suggestions?
cheers,
Aad
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to add "tokens" to a stream
Posted by Aad Nales <aa...@gmail.com>.
Yes it does,
The key bit is the part with the termAttribute...
thanks a lot,
cheers,
Aad
On Mon, Jun 7, 2010 at 4:53 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> Hey there,
> in lucene 3.0 / 2.9 the Token class has been remove / replaced with an
> Attribute based API. A TokenStream operates on Attibutes it has
> declared which are eventually accessed by the IndexWriter to create
> the inverted index. There are Attributes like TermAttribute,
> PositionIncrementAttribute and OffsetAttribute that roughly reflect
> the fields of the Token class. If you want to return an extra token /
> Term from the Attribute based API you just set the TermAttribute value
> and all other attributes you wanna use and return true from the
> TokenStream#incrementToken()
>
> something like this
>
> public boolean incrementToken() {
> if(input.incrementToken()){
> // process the tokens in the stream
> return true;
> } else if (!extraTokens.isEmtpy()){
> // return extra token
> termAttribute.setTerm(extraTokens.pop())
> return true;
> }
> return false;
>
> }
>
> does that make sense?
>
> simon
>
> On Mon, Jun 7, 2010 at 4:43 PM, Aad Nales <aa...@gmail.com> wrote:
>> Hi All,
>>
>> Years ago we implemented a Lucene solution which we are updating
>> today, and i am a bit lost on the following.
>>
>> In Lucene 1.x and 2.x it was possible to add a token in a Filter
>> simply by returning an extra Token when next was being called. What i
>> can not find is an equivalent possiblity for adding a Token in a
>> filter. The incrementToken() seem to be the logical place to implement
>> similar functionality (at least it could trigger it) but i am a bit
>> lost how to access the stream and insert an extra token. Any
>> suggestions?
>>
>> cheers,
>> Aad
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to add "tokens" to a stream
Posted by Simon Willnauer <si...@googlemail.com>.
Hey there,
in lucene 3.0 / 2.9 the Token class has been remove / replaced with an
Attribute based API. A TokenStream operates on Attibutes it has
declared which are eventually accessed by the IndexWriter to create
the inverted index. There are Attributes like TermAttribute,
PositionIncrementAttribute and OffsetAttribute that roughly reflect
the fields of the Token class. If you want to return an extra token /
Term from the Attribute based API you just set the TermAttribute value
and all other attributes you wanna use and return true from the
TokenStream#incrementToken()
something like this
public boolean incrementToken() {
if(input.incrementToken()){
// process the tokens in the stream
return true;
} else if (!extraTokens.isEmtpy()){
// return extra token
termAttribute.setTerm(extraTokens.pop())
return true;
}
return false;
}
does that make sense?
simon
On Mon, Jun 7, 2010 at 4:43 PM, Aad Nales <aa...@gmail.com> wrote:
> Hi All,
>
> Years ago we implemented a Lucene solution which we are updating
> today, and i am a bit lost on the following.
>
> In Lucene 1.x and 2.x it was possible to add a token in a Filter
> simply by returning an extra Token when next was being called. What i
> can not find is an equivalent possiblity for adding a Token in a
> filter. The incrementToken() seem to be the logical place to implement
> similar functionality (at least it could trigger it) but i am a bit
> lost how to access the stream and insert an extra token. Any
> suggestions?
>
> cheers,
> Aad
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org