You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2010/02/04 23:16:41 UTC

Re: Analyzer for stripping non alpha-numeric characters?

Transferred partially to solr-user...

Steven, thanks for the reply!

I wonder if PatternReplaceFilter can output multiple tokens?  I'd like
to progressively strip the non-alphanums, for example output:

apple!&*
apple!&
apple!
apple

On Thu, Feb 4, 2010 at 12:18 PM, Steven A Rowe <sa...@syr.edu> wrote:
> Hi Jason,
>
> Solr's PatternReplaceFilter(ts, "\\P{Alnum}+$", "", false) should work, chained after an appropriate tokenizer.
>
> Steve
>
> On 02/04/2010 at 12:18 PM, Jason Rutherglen wrote:
>> Is there an analyzer that easily strips non alpha-numeric from the end
>> of a token?
>>
>> --------------------------------------------------------------------- To
>> unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For
>> additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Analyzer for stripping non alpha-numeric characters?

Posted by Jason Rutherglen <ja...@gmail.com>.
Answering my own question... PatternReplaceFilter doesn't output
multiple tokens...

Which means messing with capture state...

On Thu, Feb 4, 2010 at 2:16 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Transferred partially to solr-user...
>
> Steven, thanks for the reply!
>
> I wonder if PatternReplaceFilter can output multiple tokens?  I'd like
> to progressively strip the non-alphanums, for example output:
>
> apple!&*
> apple!&
> apple!
> apple
>
> On Thu, Feb 4, 2010 at 12:18 PM, Steven A Rowe <sa...@syr.edu> wrote:
>> Hi Jason,
>>
>> Solr's PatternReplaceFilter(ts, "\\P{Alnum}+$", "", false) should work, chained after an appropriate tokenizer.
>>
>> Steve
>>
>> On 02/04/2010 at 12:18 PM, Jason Rutherglen wrote:
>>> Is there an analyzer that easily strips non alpha-numeric from the end
>>> of a token?
>>>
>>> --------------------------------------------------------------------- To
>>> unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For
>>> additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

Re: Analyzer for stripping non alpha-numeric characters?

Posted by Jason Rutherglen <ja...@gmail.com>.
Answering my own question... PatternReplaceFilter doesn't output
multiple tokens...

Which means messing with capture state...

On Thu, Feb 4, 2010 at 2:16 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Transferred partially to solr-user...
>
> Steven, thanks for the reply!
>
> I wonder if PatternReplaceFilter can output multiple tokens?  I'd like
> to progressively strip the non-alphanums, for example output:
>
> apple!&*
> apple!&
> apple!
> apple
>
> On Thu, Feb 4, 2010 at 12:18 PM, Steven A Rowe <sa...@syr.edu> wrote:
>> Hi Jason,
>>
>> Solr's PatternReplaceFilter(ts, "\\P{Alnum}+$", "", false) should work, chained after an appropriate tokenizer.
>>
>> Steve
>>
>> On 02/04/2010 at 12:18 PM, Jason Rutherglen wrote:
>>> Is there an analyzer that easily strips non alpha-numeric from the end
>>> of a token?
>>>
>>> --------------------------------------------------------------------- To
>>> unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For
>>> additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org