[jira] [Commented] (LUCENE-5357) Upgrade StandardTokenizer & co to latest unicode rules

["Steve Rowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840624#comment-13840624 ] 

Steve Rowe commented on LUCENE-5357:
------------------------------------

Tasks include:

* Update the TLDs acceptable in URLs and Emails (for {{UAX29URLEmailTokenizer}}) from the latest IANA Root Zone Database, using {{ant gen-tlds}}.  Test data files referring to obsolete TLDs will need to be updated to use current TLDs: {{((email.addresses,urls).from.)random.text.with.(email.address,urls).txt}}.
* Update the icu module's {{GenerateJFlexSupplementaryMacros.java}} to include supplementary character additions to JFlex grammars for new character classes {{\[:WordBreak=Single_Quote:]}}, {{\[:WordBreak=Double_Quote:]}} and {{\[:WordBreak=Hebrew_Letter:]}}.
* Update the JFlex grammars to Unicode 6.3
** Change the version in the {{%unicode}} directive in the grammar: {{%unicode 6.1}} -> {{%unicode 6.3}}
** Upgrade the UAX#29-based grammars to the Unicode 6.3 word break rules, in {{StandardTokenizerImpl.jflex}} and {{UAX29URLEmailTokenizer.jflex}}.
* Regenerate the JFlex scanners in {{lucene/analysis/common/}} via {{ant jflex}}.
* Test the new scanners against the Unicode 6.3 word break test data
** Update {{generateJavaUnicodeWordBreakTest.pl}} to handle above-BMP characters in the Unicode character database's {{ucd/auxiliary/WordBreakTest.txt}} (previous Unicode versions included only BMP characters in that file).
** Using {{generateJavaUnicodeWordBreakTest.pl}}, generate {{WordBreakTestUnicode_6_3_0.java}} under {{modules/analysis/common/src/test/org/apache/lucene/analysis/core/}}.
** Update {{TestStandardAnalyzer.java}} and {{TestUAX29URLEmailTokenizer.java}} to invoke {{WordBreakTestUnicode_6_3_0}} rather than {{WordBreakTestUnicode_6_1_0}}.
** Remove {{WordBreakTestUnicode_6_1_0.java}}.

Additional task for the 4.x backport:

* Version the JFlex grammars: 
** Copy the current implementations to *Impl40 (where 40=>4.0 is the version in which the Unicode 6.1 versions of these scanners were introduced.
** Cause the versioning tokenizer wrappers to instantiate this version when the Version c-tor param is in the range 4.0 to 4.6.
** Change the specified Unicode version in the non-versioned JFlex grammars from 6.1 to 6.3.


> Upgrade StandardTokenizer & co to latest unicode rules
> ------------------------------------------------------
>
>                 Key: LUCENE-5357
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5357
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: modules/analysis
>            Reporter: Robert Muir
>
> besides any change in data, the rules have also changed (regional indicators, better handling for hebrew, etc)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org