tokenize after filters

[Joe Calderon <ca...@gmail.com>]

 is it possible to tokenize a field on whitespace after some filters
have been applied:

ex: "A + W Root Beer"
the field uses a keyword tokenizer to keep the string together, then
it will get converted to "aw root beer" by a custom filter ive made, i
now want to split that up into 3 tokens (aw, root, beer), but seems
like you cant use a tokenizer after a filter ... so whats the best way
of accomplishing this?

thx much

--joe

RE: tokenize after filters

[Steven A Rowe <sa...@syr.edu>]

I think you want Koji Sekiguchi's Char Filters:

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters?highlight=char+filters#Char_Filters

Steve

> -----Original Message-----
> From: Joe Calderon [mailto:calderon.joe@gmail.com]
> Sent: Monday, November 02, 2009 11:25 AM
> To: solr-user@lucene.apache.org
> Subject: tokenize after filters
> 
>  is it possible to tokenize a field on whitespace after some filters
> have been applied:
> 
> ex: "A + W Root Beer"
> the field uses a keyword tokenizer to keep the string together, then
> it will get converted to "aw root beer" by a custom filter ive made, i
> now want to split that up into 3 tokens (aw, root, beer), but seems
> like you cant use a tokenizer after a filter ... so whats the best way
> of accomplishing this?
> 
> thx much
> 
> --joe

Re: tokenize after filters

[Chris Hostetter <ho...@fucit.org>]

: ex: "A + W Root Beer"
: the field uses a keyword tokenizer to keep the string together, then
: it will get converted to "aw root beer" by a custom filter ive made, i
: now want to split that up into 3 tokens (aw, root, beer), but seems

TokenFilter's can produce more tokens then they consume ... so your custom 
TOkenFilter can split the single token "aw root beer" into three tokens if 
you want it to.  (this is how WordDelimiterFilter and SYnonymFilter work)


-Hoss