You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2019/06/28 22:44:00 UTC
[jira] [Resolved] (SOLR-13580) java 13 changes to locale specific
Numeric parsing rules affect ParseNumeric UpdateProcessors when using
'local' config option - notably affects French
[ https://issues.apache.org/jira/browse/SOLR-13580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man resolved SOLR-13580.
-----------------------------
Resolution: Not A Bug
> java 13 changes to locale specific Numeric parsing rules affect ParseNumeric UpdateProcessors when using 'local' config option - notably affects French
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-13580
> URL: https://issues.apache.org/jira/browse/SOLR-13580
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
> Assignee: Hoss Man
> Priority: Major
> Labels: Java13
> Attachments: SOLR-13580.patch
>
>
> Per [JDK-8221432|https://bugs.openjdk.java.net/browse/JDK-8221432] Java13 has updated to [CLDR 35.1|http://cldr.unicode.org/] – which controls the definition of language & locale specific formatting characters – in a non-backwards compatible way due to "French" changes in [CLDR 34|http://cldr.unicode.org/index/downloads/cldr-34#TOC-Detailed-Data-Changes]
> This impacts people who use any of the "ParseNumeric" UpdateProcessors in conjunction with the "locale=fr" or "locale=fr_FR" init param and expect the (pre java13) existing behavior of treating U+00A0 (NO BREAK SPACE) as a "grouping" character (ie: between thousands and million, between millions and billions, etc...). Starting with java13 the JVM expects U+202F (NARROW NO BREAK SPACE) in it's place.
> Notably: upgrading to jdk13-ea+26 caused failures in Solr's ParsingFieldUpdateProcessorsTest which was initially had hardcoded test data that used U+00A0. ParsingFieldUpdateProcessorsTest has since been updated to account for this discrepency by modifying the test data used to determine the "expected" character for the current JVM, but there is nothing Solr or the ParseNumeric UpdateProcessors can do to help mitigate this change in behavior for end users who upgrade to java13.
> Affected users with U+00A0 characters in their incoming SolrInputDocuments will see the ParseNumeric UpdateProcessors (configured with locale=fr...) "skip" these values as unparsable, most likely resulting in a failure to index into a numeric field since the original "String" value will be left as is.
> Affected users may want to consider updating their configs to include a {{RegexReplaceProcessorFactory}} configured to strip out all whitespace characters, prior to any ParseNumeric update processors configured expect french langauge numbers
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org