You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Rowe (JIRA)" <ji...@apache.org> on 2017/10/06 16:57:00 UTC
[jira] [Updated] (LUCENE-5753) Domain lists for UAX_URL_EMAIL
analyzer are incomplete - cannot recognize ".local" among others
[ https://issues.apache.org/jira/browse/LUCENE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Rowe updated LUCENE-5753:
-------------------------------
Attachment: LUCENE-5753.patch
Patch updating TLDs for {{UAX29URLEmailTokenizer}}. Precommit and all tests pass.
{{ASCIITLD.jflex-macro}} increases from 342 to 1543 TLDs.
I had to use {{ANT_OPTS=-Xmx8g ant regenerate}} to give enough memory to JFlex. ({{-Xmx4g}} didn't work, but maybe something between 4g and 8g would - I didn't try.)
With this patch, the lucene-analyzers-common jar goes from 1.5MB to 1.6MB; I think the size increase is acceptable.
Committing shortly.
> Domain lists for UAX_URL_EMAIL analyzer are incomplete - cannot recognize ".local" among others
> -----------------------------------------------------------------------------------------------
>
> Key: LUCENE-5753
> URL: https://issues.apache.org/jira/browse/LUCENE-5753
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Reporter: Steve Merritt
> Attachments: LUCENE-5753.patch
>
>
> uax_url_email analyzer appears unable to recognize the ".local" TLD among others. Bug can be reproduced by
> curl -XGET "ADDRESS/INDEX/_analyze?text=First%20Last%20lname@section.mycorp.local&pretty&analyzer=uax_url_email"
> will parse "lname@section.my" and "corp.local" as separate tokens, as opposed to
> curl -XGET "ADDRESS/INDEX/_analyze?text=First%20Last%20lname@section.mycorp.org&pretty&analyzer=uax_url_email"
> which will recognize "lname@section.mycorp.org".
> Can this be fixed by updating to a newer version? I am running ElasticSearch 0.90.5 and whatever Lucene version sits underneath that. My suspicion is that the TLD list the analyzer relies on (http://www.internic.net/zones/root.zone, I think?) is incomplete and needs updating.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org