You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Michael J. Prichard" <mi...@mac.com> on 2006/06/16 21:20:20 UTC
indexing emails
I am working on indexing emails and want to have a "to" field. I am
currently putting all the emails on one line seperated w/ spaces...example:
michael@foo.bar john@boo.com jane@bar.com
Then i index that with a StandardAnalyzer as follows:
doc.add(new Field("to", (String) itemContent.get("to"), Field.Store.YES,
Field.Index.UN_TOKENIZED));
Question is...is this the best way to do it? I want to be able to
search for michael@foo.bar and pick out just those Documents, etc.
Thanks,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing emails
Posted by Michael Wechner <mi...@wyona.com>.
Rob Staveley (Tom) wrote:
> Having spent a lot of time getting this wrong myself in an e-mail
> indexer(!), I urge you to consider whether in your query interface you will
> need to look for mail to "john*" rather than john@boo.com, because "john*"
> may have been addressed to john@boo.net or john.smith@boo2.com. If you index
> only john@boo.com (untokenised) you will have to use a PrefixQuery to look
> for "john*", and you are liable to hit BooleanQuery.TooManyClauses problems,
> if you have more than 1024 (or BooleanQuery.getMaxClauseCount()) e-mail
> addresses in your index starting with "john".
>
> I'm trying to figure out a good design for this now for my own e-mail
> indexing application,
btw, is your code available somewhere, I mean as Open Source ;-) ?
Thanks
Michi
> considering also whether I should cater for searches
> for "*smith*". I'm coming round to the realisation that WildCardQuery and
> PrefixQuery are not great things to depend upon for getting e-mail addresses
> from an index and the right thing to do is to break the address up into
> natural tokens ('.' or '-') in one field and leave them intact in another
> field. It isn't ideal; e-mail addresses with no separator between initials
> or first names and last name still need a PrefixQuery or WildcardQuery, if
> you want to search for last names, but it does make some queries possible
> which would otherwise blow up.
>
> -----Original Message-----
> From: karl wettin [mailto:kalle@snigel.net]
> Sent: 16 June 2006 21:13
> To: java-user@lucene.apache.org
> Subject: Re: indexing emails
>
> On Fri, 2006-06-16 at 15:20 -0400, Michael J. Prichard wrote:
>
>> I am working on indexing emails and want to have a "to" field. I am
>> currently putting all the emails on one line seperated w/
>>
> spaces...example:
>
>> michael@foo.bar john@boo.com jane@bar.com
>>
>> Then i index that with a StandardAnalyzer as follows:
>>
>> doc.add(new Field("to", (String) itemContent.get("to"),
>> Field.Store.YES, Field.Index.UN_TOKENIZED));
>>
>> Question is...is this the best way to do it? I want to be able to
>> search for michael@foo.bar and pick out just those Documents, etc.
>>
>
> You can either do it as above (but you want to TOKENIZE the field) or you
> could create a new UN_TOKENIZED field for each email address.
>
> The second will require less CPU as it does not involve any lexical
> analysis. It will also create larger distance between the addresses in the
> index (see span queries and term positions).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Michael Wechner
Wyona - Open Source Content Management - Apache Lenya
http://www.wyona.com http://lenya.apache.org
michael.wechner@wyona.com michi@apache.org
+41 44 272 91 61
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing emails --> mutliple "to" emails, setting position same
Posted by "Michael J. Prichard" <mi...@mac.com>.
So I have emails with multiple recipients (of course, this is very
common). I currently put them all on the same string seperated by space
and then tokenize them with Standard Analyzer. I was looking into
SynonymAnalyzers and see that you can drop multiple tokens with the same
position. Would this be a better approach to the indexing design
mentioned above?
Thanks,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing emails
Posted by "Michael J. Prichard" <mi...@mac.com>.
This is great! I was hoping to find some people who are dealing with
this issue. I am going to try to tokenize the email addresses and see
what that does. I am going to use a StandardAnalyzer which (if I am not
mistaken) will keep the email address as is. Would I still have to use
PrefixQuery for queries such as john*?
Anyone else want to comment?
-Michael
Rob Staveley (Tom) wrote:
>Having spent a lot of time getting this wrong myself in an e-mail
>indexer(!), I urge you to consider whether in your query interface you will
>need to look for mail to "john*" rather than john@boo.com, because "john*"
>may have been addressed to john@boo.net or john.smith@boo2.com. If you index
>only john@boo.com (untokenised) you will have to use a PrefixQuery to look
>for "john*", and you are liable to hit BooleanQuery.TooManyClauses problems,
>if you have more than 1024 (or BooleanQuery.getMaxClauseCount()) e-mail
>addresses in your index starting with "john".
>
>I'm trying to figure out a good design for this now for my own e-mail
>indexing application, considering also whether I should cater for searches
>for "*smith*". I'm coming round to the realisation that WildCardQuery and
>PrefixQuery are not great things to depend upon for getting e-mail addresses
>from an index and the right thing to do is to break the address up into
>natural tokens ('.' or '-') in one field and leave them intact in another
>field. It isn't ideal; e-mail addresses with no separator between initials
>or first names and last name still need a PrefixQuery or WildcardQuery, if
>you want to search for last names, but it does make some queries possible
>which would otherwise blow up.
>
>-----Original Message-----
>From: karl wettin [mailto:kalle@snigel.net]
>Sent: 16 June 2006 21:13
>To: java-user@lucene.apache.org
>Subject: Re: indexing emails
>
>On Fri, 2006-06-16 at 15:20 -0400, Michael J. Prichard wrote:
>
>
>>I am working on indexing emails and want to have a "to" field. I am
>>currently putting all the emails on one line seperated w/
>>
>>
>spaces...example:
>
>
>>michael@foo.bar john@boo.com jane@bar.com
>>
>>Then i index that with a StandardAnalyzer as follows:
>>
>>doc.add(new Field("to", (String) itemContent.get("to"),
>>Field.Store.YES, Field.Index.UN_TOKENIZED));
>>
>>Question is...is this the best way to do it? I want to be able to
>>search for michael@foo.bar and pick out just those Documents, etc.
>>
>>
>
>You can either do it as above (but you want to TOKENIZE the field) or you
>could create a new UN_TOKENIZED field for each email address.
>
>The second will require less CPU as it does not involve any lexical
>analysis. It will also create larger distance between the addresses in the
>index (see span queries and term positions).
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: indexing emails
Posted by "Rob Staveley (Tom)" <rs...@seseit.com>.
Having spent a lot of time getting this wrong myself in an e-mail
indexer(!), I urge you to consider whether in your query interface you will
need to look for mail to "john*" rather than john@boo.com, because "john*"
may have been addressed to john@boo.net or john.smith@boo2.com. If you index
only john@boo.com (untokenised) you will have to use a PrefixQuery to look
for "john*", and you are liable to hit BooleanQuery.TooManyClauses problems,
if you have more than 1024 (or BooleanQuery.getMaxClauseCount()) e-mail
addresses in your index starting with "john".
I'm trying to figure out a good design for this now for my own e-mail
indexing application, considering also whether I should cater for searches
for "*smith*". I'm coming round to the realisation that WildCardQuery and
PrefixQuery are not great things to depend upon for getting e-mail addresses
from an index and the right thing to do is to break the address up into
natural tokens ('.' or '-') in one field and leave them intact in another
field. It isn't ideal; e-mail addresses with no separator between initials
or first names and last name still need a PrefixQuery or WildcardQuery, if
you want to search for last names, but it does make some queries possible
which would otherwise blow up.
-----Original Message-----
From: karl wettin [mailto:kalle@snigel.net]
Sent: 16 June 2006 21:13
To: java-user@lucene.apache.org
Subject: Re: indexing emails
On Fri, 2006-06-16 at 15:20 -0400, Michael J. Prichard wrote:
> I am working on indexing emails and want to have a "to" field. I am
> currently putting all the emails on one line seperated w/
spaces...example:
>
> michael@foo.bar john@boo.com jane@bar.com
>
> Then i index that with a StandardAnalyzer as follows:
>
> doc.add(new Field("to", (String) itemContent.get("to"),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
>
> Question is...is this the best way to do it? I want to be able to
> search for michael@foo.bar and pick out just those Documents, etc.
You can either do it as above (but you want to TOKENIZE the field) or you
could create a new UN_TOKENIZED field for each email address.
The second will require less CPU as it does not involve any lexical
analysis. It will also create larger distance between the addresses in the
index (see span queries and term positions).
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing emails
Posted by karl wettin <ka...@snigel.net>.
On Fri, 2006-06-16 at 15:20 -0400, Michael J. Prichard wrote:
> I am working on indexing emails and want to have a "to" field. I am
> currently putting all the emails on one line seperated w/ spaces...example:
>
> michael@foo.bar john@boo.com jane@bar.com
>
> Then i index that with a StandardAnalyzer as follows:
>
> doc.add(new Field("to", (String) itemContent.get("to"), Field.Store.YES,
> Field.Index.UN_TOKENIZED));
>
> Question is...is this the best way to do it? I want to be able to
> search for michael@foo.bar and pick out just those Documents, etc.
You can either do it as above (but you want to TOKENIZE the field) or
you could create a new UN_TOKENIZED field for each email address.
The second will require less CPU as it does not involve any lexical
analysis. It will also create larger distance between the addresses in
the index (see span queries and term positions).
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: indexing emails
Posted by John Haxby <jc...@scalix.com>.
Michael J. Prichard wrote:
> I am working on indexing emails and want to have a "to" field. I am
> currently putting all the emails on one line seperated w/
> spaces...example:
>
> michael@foo.bar john@boo.com jane@bar.com
>
> Then i index that with a StandardAnalyzer as follows:
>
> doc.add(new Field("to", (String) itemContent.get("to"),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
>
> Question is...is this the best way to do it? I want to be able to
> search for michael@foo.bar and pick out just those Documents, etc.
I took a slightly different approach. Using javamail, given a To: line
like this:
To: Fred Smith <fs...@example.com>,
=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <ke...@example.com>
I re-constructed the address list to look like this:
Fred Smith fs@example.com Keld Jørn Simonsen keld@example.com
and fed that to the analyser. I forget which analyser we eventually
settled on, but the "fred@example.com" turns into the tokens "fred"
"example" and "com". This actually gives rise to a remarkably natural
way of search for
addresses. People do things like searching for "lucene.apache.org" to
look for mail sent to the lucene lists, they search for me variously as
"jch", "john haxby" and "haxby"; they even, occasionally, search for
complete mail addresses. They all work.
The RFC2047 syntax in the example above gives one hint as to the
minefield that address parsing can be. If you look at the javamail
spec, you'll also see reference to group-syntax -- it's often seen as
undisclosed-recipients:;
but you'll also occasionally see
example-group: fred@example.com, keld@example.com;
Javamail knows how to parse these and I threw away the group name and
just indexed the messages. It might've been better to keep the group
name, but groups aren't that widely used so it probably doesn't make
much difference.
Other heads cause headaches as well. Things like the subject can be
RFC2047 encoded so you'll need to decode them. The various message-id
headers are also slightly problematic. If you're using "message-id"
and "references" and "in-reply-to" you'll need to be careful -- the
individual message-id's will need their angle brackets removed and they
really ought not to be tokenized.
It's also worth indexing *all* the message headers. People do do
searches on some odd things. I also index the raw content-type as well
-- those huge presentations can be found and deleted by searching for
"content-type:application/vnd.ms-powerpoint". Or at least I could. It
seems to be broken at the moment :-(
jch
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org