Analysis API next step: Reader->CharFilter?

[Benson Margulies <bi...@gmail.com>]

Now that we're forcing everyone to think about the Analysis API in
5.0, what do you think of making the fundamental input source be a
CharFilter, thus removing the need for instanceof-ing?

To touch a hotter potato, I also wonder about 'reset()'. In a world
where the only way to put something in there is setReader, do we need
'reset' in between setReader and incrementToken?

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Analysis API next step: Reader->CharFilter?

[Robert Muir <rc...@gmail.com>]

On Thu, Jan 9, 2014 at 9:08 AM, Benson Margulies <bi...@gmail.com> wrote:
> Now that we're forcing everyone to think about the Analysis API in
> 5.0, what do you think of making the fundamental input source be a
> CharFilter, thus removing the need for instanceof-ing?

Personally, i don't like doing that, because when we change a
parameter from a 'standard jdk' one to a custom lucene one, it makes
the API harder to grok as its more classes the user *must* wrap their
head around. On the other hand, today users only have to grok
CharFilter if they want to do CharFiltering, which is pretty expert.
Instanceofs are cheap in java, what is the benefit?

>
> To touch a hotter potato, I also wonder about 'reset()'. In a world
> where the only way to put something in there is setReader, do we need
> 'reset' in between setReader and incrementToken?

But the main issue is TokenStream: it doesnt have any concept of
Readers baked in. So there must be a way to reset state in things like
TokenFilters, too.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org