You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Rowe (JIRA)" <ji...@apache.org> on 2018/04/30 20:41:00 UTC
[jira] [Commented] (LUCENE-8278) UAX29URLEmailTokenizer is not
detecting some tokens as URL type
[ https://issues.apache.org/jira/browse/LUCENE-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459032#comment-16459032 ]
Steve Rowe commented on LUCENE-8278:
------------------------------------
Confirming there is an issue, but I don't think the spellings of "example.com" and "example.net" are the problem though; more likely this is related to an end-of-input issue. This test added to {{TestUAX29URLEmailTokenizer}} fails for me:
{noformat}
public void testExampleURLs() throws Exception {
Analyzer analyzer = new Analyzer() {
@Override protected TokenStreamComponents createComponents(String fieldName) {
return new TokenStreamComponents(new UAX29URLEmailTokenizer(newAttributeFactory()));
}};
// A trailing space allows these to succeed
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.com ", new String[]{"example.com"}, new String[]{"<URL>"});
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.net ", new String[]{"example.net"}, new String[]{"<URL>"});
// These fail
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.com", new String[]{"example.com"}, new String[]{"<URL>"});
BaseTokenStreamTestCase.assertAnalyzesTo(analyzer, "example.net", new String[]{"example.net"}, new String[]{"<URL>"});
}
{noformat}
So there is an issue here with no-scheme end-of-input URLs not being recognized as type {{<URL>}}.
> UAX29URLEmailTokenizer is not detecting some tokens as URL type
> ---------------------------------------------------------------
>
> Key: LUCENE-8278
> URL: https://issues.apache.org/jira/browse/LUCENE-8278
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Junte Zhang
> Priority: Minor
>
> We are using the UAX29URLEmailTokenizer so we can use the token types in our plugins.
> However, I noticed that the tokenizer is not detecting certain URLs as <URL> but <ALPHANUM> instead.
> Examples that are not working:
> * example.com is <ALPHANUM>
> * example.net is <ALPHANUM>
> But:
> * https://example.com is <URL>
> * as is https://example.net
> Examples that work:
> * example.ch is <URL>
> * example.co.uk is <URL>
> * example.nl is <URL>
> I have checked this JIRA, and could not find an issue. I have tested this on Lucene (Solr) 6.4.1 and 7.3.
> Could someone confirm my findings and advise what I could do to (help) resolve this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org