Use of Field(String name, TokenStream tokenStream)

[Developer Developer <de...@gmail.com>]

Hello Frens,

I am observing that a Field constructed using tokenStream i.e Filed fl  =
new Field(String name, TokenStream tokenStream)  is not converted to the
lower case when stored in the index.
The terms in the index are exactly same as those in tokenStream.
When I do a phrase search,the  PhraseQuery converts the search string to
all lower case , however it does not find any hits in the index because in
the index the terms are all in the Mixed case ( i.e as they appeared in the
tokenstream).

My questions are

is this the correct behavior of Lucene ?
If so, why do we need Tokenstream constructor  when we cannot even search on
it ?
is there a way to convert the tokens to terms while storing in the index
when using tokenstream as input to Field?

Re: Use of Field(String name, TokenStream tokenStream)

[Chris Hostetter <ho...@fucit.org>]

: I am observing that a Field constructed using tokenStream i.e Filed fl  =
: new Field(String name, TokenStream tokenStream)  is not converted to the
: lower case when stored in the index.
: The terms in the index are exactly same as those in tokenStream.
: When I do a phrase search,the  PhraseQuery converts the search string to
: all lower case , however it does not find any hits in the index because in

if this is the case, then it is because the Analyzer used by your 
QueryParser is lowercasing the tokens.  when you use the Field constructor 
that takes a TokenStream, all of the analysis for the source text should 
have alreayd been done -- if not, then use one of the other constructors 
and let the Analyser do it's thing when you add the document to your 
index.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org