You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Shawn Konopinsky <sk...@blueprint.org> on 2005/01/11 21:44:48 UTC

Token Characters

Hey There,

Wondering where I can find a list of the set of characters that the
StandardAnalyzer will tokenize on when indexing text in Lucene.

Best,
Shawn.


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Re: Token Characters

Posted by Otis Gospodnetic <ot...@yahoo.com>.
The best place to look is:
./src/java/org/apache/lucene/analysis/standard/StandardTokenizer.jj

You can see it at:
http://cvs.apache.org/viewcvs.cgi/jakarta-lucene/src/java/org/apache/lucene/analysis/standard/

Otis


--- Shawn Konopinsky <sk...@blueprint.org> wrote:

> Hey There,
> 
> Wondering where I can find a list of the set of characters that the
> StandardAnalyzer will tokenize on when indexing text in Lucene.
> 
> Best,
> Shawn.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org