Custom Tokenization of a Single Field

[Briggs <ac...@gmail.com>]

I have an index that contains a couple special fields that I need to
tokenize differently than the rest.   The case is that I basically
have a key/value pair stored as the value.  The field name is
"attribute" and it's value is "SomeValue=1.9"

I need to tokenize the value so that I can search on the field with
just "SomeValue" (i.e. attribute:SomeValue).   Is it possible to do
this?  I basically want to tokenize by "=".   The idea that someone
could search using "attribute:1.9" doesn't matter to me.

Currently, I am using Compass to do my indexing, but I might have to
rip it out if this functionality is only available in Lucene proper.

Hope this is enough info.

Thanks,

Briggs
-- 
"Conscious decisions by conscious minds are what make reality real"

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Custom Tokenization of a Single Field

[Briggs <ac...@gmail.com>]

Cool.  I just realized that compass also has an annotation value of
analyzer.  Now I'll just have to find out if you can truly have more
than one per index.

Thanks!

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Custom Tokenization of a Single Field

[Erick Erickson <er...@gmail.com>]

I *think* you want to consider writing your own analyzer that tokenizes
your special fields however you want to. Then use PerFieldAnalyzerWrapper
at both index and query time to break the stream up appropriately for this
(and only this) field..

This would give you two tokens for the attribute field, "SomeValue" and "1.9
"
so you'd even be able to express the notion of your attribute field
containing
both "SomeValue" and "1.9".

Hope this helps
Erick

On Jan 8, 2008 9:54 AM, Briggs <ac...@gmail.com> wrote:

> I have an index that contains a couple special fields that I need to
> tokenize differently than the rest.   The case is that I basically
> have a key/value pair stored as the value.  The field name is
> "attribute" and it's value is "SomeValue=1.9"
>
> I need to tokenize the value so that I can search on the field with
> just "SomeValue" (i.e. attribute:SomeValue).   Is it possible to do
> this?  I basically want to tokenize by "=".   The idea that someone
> could search using "attribute:1.9" doesn't matter to me.
>
> Currently, I am using Compass to do my indexing, but I might have to
> rip it out if this functionality is only available in Lucene proper.
>
> Hope this is enough info.
>
> Thanks,
>
> Briggs
> --
> "Conscious decisions by conscious minds are what make reality real"
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>