You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Edoardo Causarano <ed...@gmail.com> on 2017/04/21 15:20:02 UTC
A question over TokenFilters
Hi all.
I’m relatively new to Lucene, so I have a couple questions about writing custom filters.
The way I understand it, one would extend org.apache.lucene.analysis.TokenFilter and override #incrementToken to examine the current token provided by a stream token producer.
I’d like to write some logic that considers the last n seen tokens therefore I need to access this context as the filter chain is scanning the stream.
Can anyone point to an example of such a construct?
Also, how would I access and update this context keeping multithreading in mind? Actually, what is the treading model of a TokenStream, can anyone point out a good summary for it?
TIA
Best,
Edoardo
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: A question over TokenFilters
Posted by Edoardo Causarano <ed...@gmail.com>.
Hi,
thanks for your reply. In several other implementations I’ve seen this pattern of using a while(input.incrementToken()) within the filter’s incrementToken method. Is this approach recommended or are there hidden traps (eg: memory consumption, dependency on filter ordering and so on)
Best,
Edoardo
> On 21 Apr 2017, at 17:32, Ahmet Arslan <io...@yahoo.com.INVALID> wrote:
>
> Hi,
> LimitTokenCountFilter is used to index first n tokens. May be it can inspire you.
>
> Ahmet
> On Friday, April 21, 2017, 6:20:11 PM GMT+3, Edoardo Causarano <ed...@gmail.com> wrote:
> Hi all.
>
> I’m relatively new to Lucene, so I have a couple questions about writing custom filters.
>
> The way I understand it, one would extend org.apache.lucene.analysis.TokenFilter and override #incrementToken to examine the current token provided by a stream token producer.
>
> I’d like to write some logic that considers the last n seen tokens therefore I need to access this context as the filter chain is scanning the stream.
>
> Can anyone point to an example of such a construct?
>
> Also, how would I access and update this context keeping multithreading in mind? Actually, what is the treading model of a TokenStream, can anyone point out a good summary for it?
>
> TIA
>
>
> Best,
> Edoardo
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: A question over TokenFilters
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,
LimitTokenCountFilter is used to index first n tokens. May be it can inspire you.
Ahmet
On Friday, April 21, 2017, 6:20:11 PM GMT+3, Edoardo Causarano <ed...@gmail.com> wrote:
Hi all.
I’m relatively new to Lucene, so I have a couple questions about writing custom filters.
The way I understand it, one would extend org.apache.lucene.analysis.TokenFilter and override #incrementToken to examine the current token provided by a stream token producer.
I’d like to write some logic that considers the last n seen tokens therefore I need to access this context as the filter chain is scanning the stream.
Can anyone point to an example of such a construct?
Also, how would I access and update this context keeping multithreading in mind? Actually, what is the treading model of a TokenStream, can anyone point out a good summary for it?
TIA
Best,
Edoardo
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org