Apache Solr 3.1.0

[Wodek Siebor <si...@bah.com>]

I'm trying to tokenize email  and IP addresses using
StandardTokenizerFactory.
It does correctly tokenize IP address but it divides email address into two
tokens one with value before '@' and the other with value after that. 

It works correctly under Solr 1.4.1

Has anybody else tried similar thing on Solr 3.1.0 successfully or is it a
potential bug?

Thanks,
Wlodek S.



--
View this message in context: http://lucene.472066.n3.nabble.com/Apache-Solr-3-1-0-tp2866007p2866007.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Apache Solr 3.1.0

[Steven A Rowe <sa...@syr.edu>]

Hi Wodek,

UAX29URLEmailTokenizer includes all of StandardTokenizer's rules and adds rules to tokenize URLs and Emails:

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.UAX29URLEmailTokenizerFactory

Steve

> -----Original Message-----
> From: Wodek Siebor [mailto:siebor_wlodek@bah.com]
> Sent: Tuesday, April 26, 2011 11:29 AM
> To: solr-user@lucene.apache.org
> Subject: Apache Solr 3.1.0
> 
> I'm trying to tokenize email  and IP addresses using
> StandardTokenizerFactory.
> It does correctly tokenize IP address but it divides email address into
> two
> tokens one with value before '@' and the other with value after that.
> 
> It works correctly under Solr 1.4.1
> 
> Has anybody else tried similar thing on Solr 3.1.0 successfully or is it
> a
> potential bug?
> 
> Thanks,
> Wlodek S.
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Apache-
> Solr-3-1-0-tp2866007p2866007.html
> Sent from the Solr - User mailing list archive at Nabble.com.