You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Stefan Oestreicher (JIRA)" <ji...@apache.org> on 2008/08/14 13:37:47 UTC
[jira] Created: (SOLR-700) test failures because of
wrong/unexpected locale
test failures because of wrong/unexpected locale
------------------------------------------------
Key: SOLR-700
URL: https://issues.apache.org/jira/browse/SOLR-700
Project: Solr
Issue Type: Bug
Components: contrib - DataImportHandler
Affects Versions: 1.3
Reporter: Stefan Oestreicher
Priority: Minor
Attachments: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
The following unit tests fail if they're run with a german locale:
org.apache.solr.schema.LegacyDateFieldTest
org.apache.solr.schema.DateFieldTest
org.apache.solr.handler.dataimport.TestNumberFormatTransformer
The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
Patch: schema.DateField-locale.patch.txt
TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622709#action_12622709 ]
Stefan Oestreicher commented on SOLR-700:
-----------------------------------------
bq. We don't need to be paranoid about this [...]
Ok.
bq. Reading through the docs, another area I'm becoming concerned about is partial parsing of the string
I see. Interestingly this behaviour of accepting invalid strings seems to only apply to the generic formatter. I wrote some tests and one of them uses a percent formatter (I also tested currency) and that throws a ParseException.
Fixing that seems to be even easier than shown in the example :) I already have a patch ready but I guess I should open another issue for this (seems pretty seperate to me)?
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622650#action_12622650 ]
Shalin Shekhar Mangar commented on SOLR-700:
--------------------------------------------
bq. In any case the getISO3Country method of the Locale class (among others) throws an exception if it's invoked on an invalid locale. Maybe it would be best to check all available locales explicitly instead of relying on the regex, possibly by constructing a static HashMap of them?!
We don't need to be paranoid about this because if such a Locale is used for parsing, the NumberFormat#getInstance will throw a MissingResourceException for the lack of corresponding resource bundle.
Reading through the docs, another area I'm becoming concerned about is partial parsing of the string. Pehaps we should use ParsePosition and verify that the whole string was actually used. See the section on validation at http://www.ibm.com/developerworks/java/library/j-numberformat/
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622711#action_12622711 ]
Shalin Shekhar Mangar commented on SOLR-700:
--------------------------------------------
bq. Fixing that seems to be even easier than shown in the example I already have a patch ready but I guess I should open another issue for this (seems pretty seperate to me)?
By all means, go ahead :)
I will commit this one now.
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: handler.dataimport.NumberFormatTransformer-locale.patch..txt
based on Shalins suggestion to add an attribute for the locale I prepared another patch implementing those changes.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: schema.DateField-locale.patch.txt
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622561#action_12622561 ]
Shalin Shekhar Mangar commented on SOLR-700:
--------------------------------------------
This is great, thanks Stefan!
IMO, one JIRA issue should deal with one task. Do you mind opening another issue for the DateField and LegacyDateField bug and attach it's patch there? I shall commit your NumberFormatTransformer enhancement patch, edit this issue's description and update the wiki docs.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622564#action_12622564 ]
Stefan Oestreicher commented on SOLR-700:
-----------------------------------------
done: SOLR-701.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-700:
---------------------------------------
Summary: NumberFormatTransformer should have configurable locales (was: test failures because of wrong/unexpected locale)
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622530#action_12622530 ]
Shalin Shekhar Mangar commented on SOLR-700:
--------------------------------------------
Thanks Stefan.
I was thinking about having a locale attribute on each field element in data-config.xml. We should modify the NumberFormatTransformer#transformRow method to read each field's locale attribute. Every field's attributes are exposed as a map, so we should have:
{code}
// Each locale must be of format lang:country e.g. en-US
String locale = fld.get("locale");
// Create a locale object based on language and country and pass it to the process method
{code}
That way locale will be configurable on a per-field basis and no EntityProcessor would need to be modified.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622603#action_12622603 ]
Stefan Oestreicher commented on SOLR-700:
-----------------------------------------
Nice. I also stumbled upon the fact that the java docs don't say anything about invalid locales. I quickly tested that and interestingly the DateFormatSymbols instance returned "." as decimal and "," as grouping separator. I would've expected the default locale (which is de-AT in my case) to be used but obviously it did not. In any case the getISO3Country method of the Locale class (among others) throws an exception if it's invoked on an invalid locale. Maybe it would be best to check all available locales explicitly instead of relying on the regex, possibly by constructing a static HashMap of them?!
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar resolved SOLR-700.
----------------------------------------
Resolution: Fixed
Committed revision 686176.
Thanks Stefan!
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-700:
---------------------------------------
Fix Version/s: 1.3
Assignee: Shalin Shekhar Mangar
Description:
NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
{code:xml}
<field column="myNumber" formatStyle="number" locale="de-DE" />
{code}
was:
The following unit tests fail if they're run with a german locale:
org.apache.solr.schema.LegacyDateFieldTest
org.apache.solr.schema.DateFieldTest
org.apache.solr.handler.dataimport.TestNumberFormatTransformer
The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
Patch: schema.DateField-locale.patch.txt
TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
Issue Type: Improvement (was: Bug)
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: handler.dataimport.NumberFormatTransformer-locale.patch..txt
I see, that's certainly more flexible. I modified the patch.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) NumberFormatTransformer should have
configurable locales
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-700:
---------------------------------------
Attachment: SOLR-700.patch
A new patch (SOLR-700.patch) based on Stefan's work with the following changes:
# Changed the locale parsing code to use pre-compiled regex.
# Throw an exception if an invalid value (which does not match the regex) is specified. Interestingly, the java.util.Locale class's docs do not say anything about invalid language or countries.
# Modified TestNumberFormatTransformer to demonstrate the fix and enhancement.
I will commit this shortly.
> NumberFormatTransformer should have configurable locales
> --------------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Assignee: Shalin Shekhar Mangar
> Priority: Minor
> Fix For: 1.3
>
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt, SOLR-700.patch
>
>
> NumberFormatTransformer uses a NumberFormatter which relies on the system locale. This makes it impossible to use NumberFormatTransformer with data whose locale does not match the system locale.
> TestNumberFormatTransformer fails on some locales for similar reasons because the grouping symbol differs in different locales.
> This issue adds a locale attribute for NumberFormatTransformer which allows the user to specify the locale which should be used for formatting. The locale must be specified as land-country e.g. en-US
> {code:xml}
> <field column="myNumber" formatStyle="number" locale="de-DE" />
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: handler.dataimport.NumberFormatTransformer-locale.patch..txt
minor update using the LOCALE constant instead of literal String.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-700) test failures because of
wrong/unexpected locale
Posted by "Stefan Oestreicher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Oestreicher updated SOLR-700:
------------------------------------
Attachment: handler.dataimport.NumberFormatTransformer-locale.patch..txt
setting the locale to the default locale just to have it changed afterwards if the parameter exists is not very nice I guess, so I changed that as well.
Furthermore the user should probably be notified if he supplies an invalid locale, but I'm unsure how to proceed in this case, so I leave this for now as it is.
> test failures because of wrong/unexpected locale
> ------------------------------------------------
>
> Key: SOLR-700
> URL: https://issues.apache.org/jira/browse/SOLR-700
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Reporter: Stefan Oestreicher
> Priority: Minor
> Attachments: handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.NumberFormatTransformer-locale.patch..txt, handler.dataimport.TestNumberFormatTransformer-locale.patch.txt, schema.DateField-locale.patch.txt
>
>
> The following unit tests fail if they're run with a german locale:
> org.apache.solr.schema.LegacyDateFieldTest
> org.apache.solr.schema.DateFieldTest
> org.apache.solr.handler.dataimport.TestNumberFormatTransformer
> The DateField tests fail because the expected decimal format uses "." as separator but the actual result is formatted using the "," separator.
> This happens because the ISO8601CanonicalDateFormat class which is defined inside the DateField class doesn't specify the US locale when creating the formatter used for milliseconds.
> Patch: schema.DateField-locale.patch.txt
> TestNumberFormatTransformer failed because handler.dataimport.NumberFormatTransformer uses a NumberFormatter which relies on the system locale. But I think in this case it's intentional, so I modified the test case to use the grouping separator of the system locale.
> Patch: handler.dataimport.TestNumberFormatTransformer-locale.patch.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.