Token or not Token, PerFieldAnalyzer

[Florian Sauvin <fl...@archive.org>]

Hello,

When indexing a field, we have the choice of tokenizing it or not. I
have a custom analyzer that contains a tokenizer... does it mean that
if the boolean token is set to false, the analyzer is not applied on
the field content?

Everywhere in the documentation (and it seems logical) you say to use
the same analyzer for indexing and querying... how is this handled on
not tokenized fields?

In my case, I have certain fields on which I want the tokenization and
anlysis and everything to happen... but on other fields, I just want to
index the content as it is (no alterations at all) and not analyze at
query time... is that possible?

--

Florian


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Token or not Token, PerFieldAnalyzer

[Doug Cutting <cu...@apache.org>]

Florian Sauvin wrote:
> Everywhere in the documentation (and it seems logical) you say to use
> the same analyzer for indexing and querying... how is this handled on
> not tokenized fields?

Imperfectly.

The QueryParser knows nothing about the index, so it does not know which 
fields were tokenized and which were not.  Moreover, even the index does 
not know this, since you can freely intermix tokenized and untokenized 
values in a single field.

> In my case, I have certain fields on which I want the tokenization and
> anlysis and everything to happen... but on other fields, I just want to
> index the content as it is (no alterations at all) and not analyze at
> query time... is that possible?

It is very possible.  A good way to handle this is to use 
PerFieldAnalyzerWrapper.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org