You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by GitBox <gi...@apache.org> on 2021/05/31 13:57:03 UTC
[GitHub] [thrift] aaronmjones opened a new pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
aaronmjones opened a new pull request #2401:
URL: https://github.com/apache/thrift/pull/2401
Add unit test for system locale with thousands separator comma
Install en_US.UTF-8 locale in Dockerfile
The global locale affects integer formatting. Some LC_NUMERIC locales specify a thousands separator "," which would break integer formatting. Add a unit test that sets the global locale to "en_US.UTF-8" (which uses a comma thousands separator). Fix the integer formatting by imbue'ing the "C" locale in ostringstream.
<!-- We recommend you review the checklist/tips before submitting a pull request. -->
- [x] Did you create an [Apache Jira](https://issues.apache.org/jira/projects/THRIFT/issues/) ticket? (not required for trivial changes)
- [x] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
- [x] Did you squash your changes to a single commit? (not required, but preferred)
- [x] Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
- [x] If your change does not involve any code, include `[skip ci]` anywhere in the commit message to free up build resources.
<!--
The Contributing Guide at:
https://github.com/apache/thrift/blob/master/CONTRIBUTING.md
has more details and tips for committing properly.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-894140312
Dear @aaronmjones,
our CI was silently not running the tests previously, so your Windows test was never executed. In a newer PR I have fixed that, but it seems that this breaks your test. Could you kindly take a look at https://ci.appveyor.com/project/ApacheSoftwareFoundation/thrift/builds/40271635/job/3dser86k51a8eco2?fullLog=true where it says
```
Test command: C:\projects\build\MSVC2015\x86\bin\Debug\UnitTests.exe
Test timeout computed to be: 300
Running 57 test cases...
unknown location(0): fatal error: in "ToStringTest/locale_en_US_int_to_string": f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c(89) : Assertion failed: ploc->_Mbcurmax == 1 || ploc->_Mbcurmax == 2
C:\projects\thrift\lib\cpp\test\ToStringTest.cpp(44): last checkpoint: "locale_en_US_int_to_string" entry.
unknown location(0): fatal error: in "ToStringTest/locale_de_DE_floating_point_to_string": f:\dd\vctools\crt\crtw32\stdcpp\xmbtowc.c(89) : Assertion failed: ploc->_Mbcurmax == 1 || ploc->_Mbcurmax == 2
C:\projects\thrift\lib\cpp\test\ToStringTest.cpp(53): last checkpoint: "locale_de_DE_floating_point_to_string" entry.
An error message from getaddrinfo on the console is expected:
Thrift: Thu Aug 5 19:53:55 2021 getaddrinfo() -> 11001; No such host is known.
*** 2 failures are detected in the test module "thrift"
Test #381: UnitTests ........................***Failed 3.59 sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-857752327
Build errors are all unrelated, merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-905715309
Implemented in e664ac49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-854547134
Other than the additional test for `to_string()`, this looks good and valuable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] aaronmjones commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
aaronmjones commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-855263748
> Would it be possible to add also a small test directly to `lib/cpp/test/ToStringTest.cpp` to see more transparently what case will fail? Is this only about JSON or are all number converters affected (I assume the latter is true)?
Yeah, ToStringTest.cpp looks like a better place for this than JSONProtoTest.cpp. The problem is about number formatting and not JSON in particular. I'll add unit tests for integer and float string formatting to ToStringTest.cpp and remove the changes to JSONProtocolTest. The integer string formatting test will set the global locale to "en_US.UTF-8" which uses comma for a thousands separator. The float string formatting test will set the global locale to "de_DE.UTF-8" which uses a decimal comma -- not a decimal point.
Before changes to TToString.h, the tests would fail like:
```
# ./UnitTests
Running 57 test cases...
/thrift/src/lib/cpp/test/ToStringTest.cpp(52): error: in "ToStringTest/locale_en_US_int_to_string": check to_string(1000000) == "1000000" has failed [1,000,000 != 1000000]
/thrift/src/lib/cpp/test/ToStringTest.cpp(62): error: in "ToStringTest/locale_de_DE_float_to_string": check to_string(1.2) == "1.2" has failed [1,2 != 1.2]
```
I'll have the changes ready soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] aaronmjones commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
aaronmjones commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-896141365
That assertion happens in the std::locale() constructor and was also reported at https://developercommunity.visualstudio.com/t/stdlocale-l2-en-usutf-8-broken-crashes-now/330322. That link says it was fixed in VS2019 16.9.
Maybe there's a way to avoid the assertion with the version of VS the Windows CI is using (is it VS2015?). But reading the documentation of std::locale(), it can throw:
>std::runtime_error if the operating system has no locale named std_name or if std_name is a null pointer.
I modified the ubuntu Dockerfiles to install the locales used by the unit tests ("en-US.UTF-8" and de-DE.UTF-8"). I don't know how to do the same on Windows in order to guarantee std::locale() does not throw.
How do you feel about enabling those 2 unit tests only on Linux platforms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau merged pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau merged pull request #2401:
URL: https://github.com/apache/thrift/pull/2401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] emmenlau commented on pull request #2401: THRIFT-3840: C++ TJSONProtocol still using locale dependent formatting
Posted by GitBox <gi...@apache.org>.
emmenlau commented on pull request #2401:
URL: https://github.com/apache/thrift/pull/2401#issuecomment-897391440
Yes, maybe you are right and this is the way to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org