You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Luis Cappa Banda <lu...@gmail.com> on 2013/07/30 16:53:19 UTC
Email regular expression.
Hello everyone!
Unfortunately I have to search all E-mail addresses found in a text field
from each document. I've been reading for a while how to use RegExp's in
Solr, but after trying some of them they didn't work. I've noticed that
Lucene RegExp syntax sometimes is very different from the classic RegExp
syntax, so that may be the reason why they didn't work for me, and maybe
someone more expert can help me.
The syntax is the following:
*E-mail: *
text:/[a-z0-9_\|-]+(\.[a-z0-9_\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]|)*\.([a-z]{2,4})/
Thank you very much in advance!
Best regards,
--
- Luis Cappa
Re: Email regular expression.
Posted by Luis Cappa Banda <lu...@gmail.com>.
I´ve been re-reading about that in older solr-mail-list messages, and it
seems that a query like 'field:*' implies that internally the whole terms
indexed are checked one by one even if they are some caches filled for that
field. That make reasonable my poor performance in the past.
However, it may be possible to create a field called 'flagEmails' that will
be true if the field 'emails' is filled via UAX29URLEmailTokenizerFactory.
Does anyone implemented during index-time this kind of behavior? Is it
possible?
Regards,
2013/7/30 Luis Cappa Banda <lu...@gmail.com>
> I´ve tried this kind of queries in the past but I detected that they have
> a poor performance and that they are incredibly slow. But it´s just my
> experience, maybe someone can share with us any other opinion.
>
>
> 2013/7/30 Raymond Wiker <rw...@gmail.com>
>
>> On Jul 30, 2013, at 22:05 , Luis Cappa Banda <lu...@gmail.com> wrote:
>> > Anyway, I still need to do a query like the following to retrieve those
>> > documents with at least one E-mail detected:
>> >
>> > http://localhost:8080/mysolr/select?q=emails:[* TO
>> > *]&start=0&rows=10&sort=mydate desc
>>
>> Can't you just use emails:* ?
>>
>>
>>
>
>
> --
> - Luis Cappa
>
--
- Luis Cappa
Re: Email regular expression.
Posted by Luis Cappa Banda <lu...@gmail.com>.
I´ve tried this kind of queries in the past but I detected that they have a
poor performance and that they are incredibly slow. But it´s just my
experience, maybe someone can share with us any other opinion.
2013/7/30 Raymond Wiker <rw...@gmail.com>
> On Jul 30, 2013, at 22:05 , Luis Cappa Banda <lu...@gmail.com> wrote:
> > Anyway, I still need to do a query like the following to retrieve those
> > documents with at least one E-mail detected:
> >
> > http://localhost:8080/mysolr/select?q=emails:[* TO
> > *]&start=0&rows=10&sort=mydate desc
>
> Can't you just use emails:* ?
>
>
>
--
- Luis Cappa
Re: Email regular expression.
Posted by Raymond Wiker <rw...@gmail.com>.
On Jul 30, 2013, at 22:05 , Luis Cappa Banda <lu...@gmail.com> wrote:
> Anyway, I still need to do a query like the following to retrieve those
> documents with at least one E-mail detected:
>
> http://localhost:8080/mysolr/select?q=emails:[* TO
> *]&start=0&rows=10&sort=mydate desc
Can't you just use emails:* ?
Re: Email regular expression.
Posted by Luis Cappa Banda <lu...@gmail.com>.
Hello guys,
Hey, I think I´ve found how to do this just adding a filter. Just for
anyone´s curiosity:
<fieldType name="emails" class="solr.TextField" sortMissingLast="true"
omitNorms="true">
<analyzer>
<tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
<filter class="solr.TypeTokenFilterFactory" types="email_type.txt"
useWhitelist="true"/>
</analyzer>
</fieldType>
Anyway, I still need to do a query like the following to retrieve those
documents with at least one E-mail detected:
http://localhost:8080/mysolr/select?q=emails:[* TO
*]&start=0&rows=10&sort=mydate desc
And I don´t like it, to be honest,
Regards,
2013/7/30 Luis Cappa Banda <lu...@gmail.com>
> Hello, Jack, Steve,
>
> Thank you for your answers. I´ve never used UAX29URLEmailTokenizerFactory,
> but I´ve read about it before trying RegExp´s queries. As far as I know, UAX29URLEmailTokenizerFactory
> allows to tokenize an entry text value into patterns that match URLs,
> E-mails, etc. Reading the documentation I haven´t found any way to select
> just E-mail patterns, not URL ones, for example. I feel that it may have
> sense to specify one or multiple patterns in a configuration file to be
> setted during the Tokenizer definition in the schema.xml, but I found
> nothing.
>
> I´ve just want to retrieve those documents indexed where they appear at
> least one E-mail inside de text. However, even using UAX29URLEmailTokenizerFactory,
> and suposing that I store that E-mail data in a field called 'emails' (I
> feel creative, hehe), a query like the following appears to be... dirty:
>
> http://localhost:8080/mysolr/select?q=emails:[* TO
> *]&start=0&rows=10&sort=mydate desc
>
> What do you think about?
>
> And Andy... I know many RegExps to find E-mail patterns in a text - that
> wasn´t my question, and of course there is no perfect one. However, Lucene
> RegExp syntax is different from classic RegExp one, so is not as easy as
> copy & paste any RegExps and, voilá! E-mails everywhere.
>
> Thank you very much in advance,
>
> Best regards,
>
>
>
>
>
> 2013/7/30 Jack Krupansky <ja...@basetechnology.com>
>
>> Just use the UAX29URLEmailTokenizerFactory, which recognizes email
>> addresses.
>>
>> Any particular reason that you're trying to reinvent the wheel?
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Luis Cappa Banda
>> Sent: Tuesday, July 30, 2013 10:53 AM
>> To: solr-user@lucene.apache.org
>> Subject: Email regular expression.
>>
>>
>> Hello everyone!
>>
>> Unfortunately I have to search all E-mail addresses found in a text field
>> from each document. I've been reading for a while how to use RegExp's in
>> Solr, but after trying some of them they didn't work. I've noticed that
>> Lucene RegExp syntax sometimes is very different from the classic RegExp
>> syntax, so that may be the reason why they didn't work for me, and maybe
>> someone more expert can help me.
>>
>> The syntax is the following:
>>
>> *E-mail: *
>>
>> text:/[a-z0-9_\|-]+(\.[a-z0-9_**\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]**
>> |)*\.([a-z]{2,4})/
>>
>> Thank you very much in advance!
>>
>> Best regards,
>>
>> --
>> - Luis Cappa
>>
>
>
>
> --
> - Luis Cappa
>
--
- Luis Cappa
Re: Email regular expression.
Posted by Luis Cappa Banda <lu...@gmail.com>.
Hello, Jack, Steve,
Thank you for your answers. I´ve never used UAX29URLEmailTokenizerFactory,
but I´ve read about it before trying RegExp´s queries. As far as I
know, UAX29URLEmailTokenizerFactory
allows to tokenize an entry text value into patterns that match URLs,
E-mails, etc. Reading the documentation I haven´t found any way to select
just E-mail patterns, not URL ones, for example. I feel that it may have
sense to specify one or multiple patterns in a configuration file to be
setted during the Tokenizer definition in the schema.xml, but I found
nothing.
I´ve just want to retrieve those documents indexed where they appear at
least one E-mail inside de text. However, even using
UAX29URLEmailTokenizerFactory,
and suposing that I store that E-mail data in a field called 'emails' (I
feel creative, hehe), a query like the following appears to be... dirty:
http://localhost:8080/mysolr/select?q=emails:[* TO
*]&start=0&rows=10&sort=mydate desc
What do you think about?
And Andy... I know many RegExps to find E-mail patterns in a text - that
wasn´t my question, and of course there is no perfect one. However, Lucene
RegExp syntax is different from classic RegExp one, so is not as easy as
copy & paste any RegExps and, voilá! E-mails everywhere.
Thank you very much in advance,
Best regards,
2013/7/30 Jack Krupansky <ja...@basetechnology.com>
> Just use the UAX29URLEmailTokenizerFactory, which recognizes email
> addresses.
>
> Any particular reason that you're trying to reinvent the wheel?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Luis Cappa Banda
> Sent: Tuesday, July 30, 2013 10:53 AM
> To: solr-user@lucene.apache.org
> Subject: Email regular expression.
>
>
> Hello everyone!
>
> Unfortunately I have to search all E-mail addresses found in a text field
> from each document. I've been reading for a while how to use RegExp's in
> Solr, but after trying some of them they didn't work. I've noticed that
> Lucene RegExp syntax sometimes is very different from the classic RegExp
> syntax, so that may be the reason why they didn't work for me, and maybe
> someone more expert can help me.
>
> The syntax is the following:
>
> *E-mail: *
>
> text:/[a-z0-9_\|-]+(\.[a-z0-9_**\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]**
> |)*\.([a-z]{2,4})/
>
> Thank you very much in advance!
>
> Best regards,
>
> --
> - Luis Cappa
>
--
- Luis Cappa
Re: Email regular expression.
Posted by Jack Krupansky <ja...@basetechnology.com>.
Just use the UAX29URLEmailTokenizerFactory, which recognizes email
addresses.
Any particular reason that you're trying to reinvent the wheel?
-- Jack Krupansky
-----Original Message-----
From: Luis Cappa Banda
Sent: Tuesday, July 30, 2013 10:53 AM
To: solr-user@lucene.apache.org
Subject: Email regular expression.
Hello everyone!
Unfortunately I have to search all E-mail addresses found in a text field
from each document. I've been reading for a while how to use RegExp's in
Solr, but after trying some of them they didn't work. I've noticed that
Lucene RegExp syntax sometimes is very different from the classic RegExp
syntax, so that may be the reason why they didn't work for me, and maybe
someone more expert can help me.
The syntax is the following:
*E-mail: *
text:/[a-z0-9_\|-]+(\.[a-z0-9_\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]|)*\.([a-z]{2,4})/
Thank you very much in advance!
Best regards,
--
- Luis Cappa
Re: Email regular expression.
Posted by Andy Lester <an...@petdance.com>.
On Jul 30, 2013, at 9:53 AM, Luis Cappa Banda <lu...@gmail.com> wrote:
> The syntax is the following:
>
> *E-mail: *
> text:/[a-z0-9_\|-]+(\.[a-z0-9_\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]|)*\.([a-z]{2,4})/
Please note that the question of "How do I write a regex to match an email address" is one of the most discussed on the Internet. Googling for "email address regular expression" will give you many many many many hits discussing how to do it, and lots of hotly-contested debates. The topic is not nearly as simple as you might think at first glance.
There is no "right" way to do it. Every approach you take will involve tradeoffs. Read up on this already well-discussed topic and decide what answer is best for you in your case.
xoa
--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
Re: Email regular expression.
Posted by Steve Rowe <sa...@gmail.com>.
Luis, do you know about UAX29URLEmailTokenizerFactory?:
<http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.UAX29URLEmailTokenizerFactory>
On Jul 30, 2013, at 10:53 AM, Luis Cappa Banda <lu...@gmail.com> wrote:
> Hello everyone!
>
> Unfortunately I have to search all E-mail addresses found in a text field
> from each document. I've been reading for a while how to use RegExp's in
> Solr, but after trying some of them they didn't work. I've noticed that
> Lucene RegExp syntax sometimes is very different from the classic RegExp
> syntax, so that may be the reason why they didn't work for me, and maybe
> someone more expert can help me.
>
> The syntax is the following:
>
> *E-mail: *
> text:/[a-z0-9_\|-]+(\.[a-z0-9_\|-]|)*@[a-z0-9-]|(\.[a-z0-9-]|)*\.([a-z]{2,4})/
>
> Thank you very much in advance!
>
> Best regards,
>
> --
> - Luis Cappa