ICU library version - was: IBM DFDL is upgrading ICU level

["Beckerle, Mike" <mb...@owlcyberdefense.com>]

DFDL implementors at IBM have noticed some issues with the ICU library worth noting.
There is an ICU pull request to fix, targeted at ICU version 70.1.

________________________________
From: ....
Subject: IBM DFDL is upgrading ICU level

Hi Mike

We are moving up the level of ICU that IBM DFDL is built with, as it is still on 51 which is out-of-support. We are trying 68.x.
In the process we found several of our regression tests failed due to behaviour changes in lax decimal/calendar processing.
If you recall we deliberately changed the DFDL 1.0 spec to make lax implementation-dependent/defined (I forget which).
We've analysed the differences and most are to do with bug fixes or other changes that are acceptable or benign or we don't think any of our customer will hit.
However, we got 400 failures in our Java version which didn't appear in our C version.
This looks to be have been caused by a regression somewhere, we think back in 62 - see https://unicode-org.atlassian.net/browse/ICU-20425.
ICU have accepted there is a problem and the fix is in PR https://github.com/unicode-org/icu/pull/1726 which is targeted at 70.1.
Letting you know as moving to 70.1 and higher might therefore cause a Daffodil behaviour change.

Regards

Steve Hanson

IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL<http://www.ibm.com/developerworks/library/se-dfdl/index.html>
Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/>

RE: ICU library version - was: IBM DFDL is upgrading ICU level

["Interrante, John A (GE Research, US)" <Jo...@ge.com>]

The fix to ICU's Java code changes only a single line and adds a new test case.  But that single line was causing 400 test cases in IBM DFDL to fail because it'd made ICU's Java code behave differently than ICU's C code (ouch).  The difference seems to be mostly rounding of real numbers, e.g., df.format(1235.5) with pattern "##00" was producing "1236.0" instead of the expected "1236".  Ideally the PR will be merged, ICU Java will produce "1236" like ICU C does, and IBM's 400 test cases will pass again even though some of Daffodil's test cases may or may not even notice this difference in behavior (it'll depend on whether any test were using this kind of pattern on real numbers).

-----Original Message-----
From: Beckerle, Mike <mb...@owlcyberdefense.com> 
Sent: Wednesday, May 19, 2021 8:54 AM
To: dev@daffodil.apache.org
Subject: EXT: ICU library version - was: IBM DFDL is upgrading ICU level

DFDL implementors at IBM have noticed some issues with the ICU library worth noting.
There is an ICU pull request to fix, targeted at ICU version 70.1.

________________________________
From: ....
Subject: IBM DFDL is upgrading ICU level

Hi Mike

We are moving up the level of ICU that IBM DFDL is built with, as it is still on 51 which is out-of-support. We are trying 68.x.
In the process we found several of our regression tests failed due to behaviour changes in lax decimal/calendar processing.
If you recall we deliberately changed the DFDL 1.0 spec to make lax implementation-dependent/defined (I forget which).
We've analysed the differences and most are to do with bug fixes or other changes that are acceptable or benign or we don't think any of our customer will hit.
However, we got 400 failures in our Java version which didn't appear in our C version.
This looks to be have been caused by a regression somewhere, we think back in 62 - see https://unicode-org.atlassian.net/browse/ICU-20425.
ICU have accepted there is a problem and the fix is in PR https://github.com/unicode-org/icu/pull/1726 which is targeted at 70.1.
Letting you know as moving to 70.1 and higher might therefore cause a Daffodil behaviour change.

Regards

Steve Hanson

IBM Hybrid Integration, Hursley, UK
Architect, IBM DFDL<http://www.ibm.com/developerworks/library/se-dfdl/index.html>
Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/>