Document design and analyzer questions?

["Michael J. Prichard" <mi...@mac.com>]

Hello,

I will try this again....

I am working on a system that will index emails and their attachments.  
I have all the pieces working that parse the documents and I am now 
working on the actual indexing part.  I would like to have synonym 
searching as well.

Question is two fold.  One, here is the layout I was thinking:

ItemID:  Unindexed, Stored
ItemGUID: Unindexed, Stored
ItemType: Indexed, Stored
ItemDate: Indexed, Unstored
ItemKeywords: Indexed, Unstored
ItemName: Indexed, Unstored
ItemContent: Indexed, Unstored
ItemFrom: Indexed, Unstored
ItemTo: Indexed, Unstored
ItemCC: Indexed, Unstored
ItemBCC: Indexed, Unstored

Searches will be combinations of To, From, CC, Date and Content.   Does 
this look ok?

Also, any recommendations on what analyzer to use?  I was thinking the 
synonym analyzer based on the one in the Lucene in Action book.

Thanks,
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Document design and analyzer questions?

[Chris Hostetter <ho...@fucit.org>]

: I will have millions of entries in my index.  Would storing them cause
: any performance issues?

only testing will tell ... but generally speaking i don't think stored
affect query performance very much -- just disk usage.

: >another important thing you should consider is field norms: they don't
: >make sense for most date fields or numeric fields, or fields where the
: >length is fairly irrelevant (ie: email addresses, guids, document types)

: What do you mean?

Check the archives and/or LIA for what a field norm is .. then look at
using Field.setOmitNorm to disable them for fields where you don't
want/need them.


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Document design and analyzer questions?

["Michael J. Prichard" <mi...@mac.com>]

Hey Chris,

Thanks for the response.

Chris Hostetter wrote:

>: Question is two fold.  One, here is the layout I was thinking:
>
>my rule of thumb: if a field is going to contain less then a few dozen
>bytes (ie: a date, an email address, etc) you might as well store it ...
>it will make your life easier when looking at your results.
>
>  
>
I will have millions of entries in my index.  Would storing them cause 
any performance issues?

>another important thing you should consider is field norms: they don't
>make sense for most date fields or numeric fields, or fields where the
>length is fairly irrelevant (ie: email addresses, guids, document types)
>
>  
>
What do you mean?

>: Also, any recommendations on what analyzer to use?  I was thinking the
>: synonym analyzer based on the one in the Lucene in Action book.
>
>you are probably going to want to use PerFieldAnalyzer so you can use a
>differnet analyzer for the fields that store email addresses then the
>analyzer you use for the body text ... if i'm searching for
>emails from "bob@car.com" i dont want it matching on emails from
>"bob@automobile.com"
>
>  
>
Sounds good.  I guess I could just use a StandardAnalyzer on these fields.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Document design and analyzer questions?

[Chris Hostetter <ho...@fucit.org>]

: Question is two fold.  One, here is the layout I was thinking:

my rule of thumb: if a field is going to contain less then a few dozen
bytes (ie: a date, an email address, etc) you might as well store it ...
it will make your life easier when looking at your results.

another important thing you should consider is field norms: they don't
make sense for most date fields or numeric fields, or fields where the
length is fairly irrelevant (ie: email addresses, guids, document types)

: Also, any recommendations on what analyzer to use?  I was thinking the
: synonym analyzer based on the one in the Lucene in Action book.

you are probably going to want to use PerFieldAnalyzer so you can use a
differnet analyzer for the fields that store email addresses then the
analyzer you use for the body text ... if i'm searching for
emails from "bob@car.com" i dont want it matching on emails from
"bob@automobile.com"



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org