You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Beckerle, Mike" <mb...@owlcyberdefense.com> on 2020/10/14 19:15:17 UTC
issue with calendarPattern with SSSSSS not unparsing to 000000
I did not find JIRA tickets for this issue. Just want to see if anyone has insight/workaround for this.
This problem occurs for Daffodil 2.4.0 and 2.7.0
I have a schema with
<xs:element
name="TimestampNano"
type="xs:dateTime"
dfdl:calendarPattern="yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZZZZ" />
Notice the 6 S in SSSSSS.
This parses this fine:
2003-08-24T05:14:15.000003-07:00
producing this XML Infoset:
<TimestampNano>2003-08-24T05:14:15.000003-07:00</TimestampNano>
So it appears that Daffodil supports microseconds. But that unparses to:
2003-08-24T05:14:15.000-07:00
This appears to be a bug. DFDL Spec says:
When the number of "S" symbols in a pattern exceeds the supported accuracy, excess fractional seconds are truncated from the right (not rounded) when parsing, and zeros are added to the right when unparsing. For example, a DFDL processor allows up to six "S" symbols and has millisecond accuracy; for pattern "ss.SSSSSS", data "12.345678" would be parsed into Infoset xs:time "00:00:12:345", which would be unparsed into data "12.345000".
This suggests I should be getting all 6 characters for SSSSSS, even if
it loses precision of the 000003 I should get 000000.
But I should not be getting just 000.
A bug? Suggestions for how to work around?
Re: issue with calendarPattern with SSSSSS not unparsing to 000000
Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Some pilot error here. I did find two places with ss.SSS and another with ss.SSSSSS, so by changing to SSSSSS everywhere now I get 6 digits, and per that ICU4J description, times like 32.000003 unparse to 32.000000, as expected.
-mikeb
________________________________
From: Steve Lawrence <st...@gmail.com>
Sent: Wednesday, October 14, 2020 3:41 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: issue with calendarPattern with SSSSSS not unparsing to 000000
I'm pretty sure we just use ICU4J for unparsing times.
We do have custom code to parse the XML infoset dateTime in
DFDLCalendarConversion.scala. So it's possible we're just parsing that
string from the infoset incorrectly. Skimming that code to see if
there's something obviously wrong, I see this comment in timePartFromString:
// ICU only supports integer milliseconds precision, which means we can
// only support at most 3 digits from the milliseconds field
And in timePartToString I see that we get the subsecond part and pad
three 0's when creating the infoset.
Seems like ICU doesn't support that much precision, and I'd guess it
doesn't padd to the number of S's in the pattern.
Might be another case where ICU behavior and the DFDL specification
don't agree.
On 10/14/20 3:15 PM, Beckerle, Mike wrote:
> I did not find JIRA tickets for this issue. Just want to see if anyone has insight/workaround for this.
>
> This problem occurs for Daffodil 2.4.0 and 2.7.0
>
> I have a schema with
>
> <xs:element
> name="TimestampNano"
> type="xs:dateTime"
> dfdl:calendarPattern="yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZZZZ" />
>
> Notice the 6 S in SSSSSS.
>
> This parses this fine:
>
> 2003-08-24T05:14:15.000003-07:00
>
> producing this XML Infoset:
>
> <TimestampNano>2003-08-24T05:14:15.000003-07:00</TimestampNano>
>
> So it appears that Daffodil supports microseconds. But that unparses to:
>
> 2003-08-24T05:14:15.000-07:00
>
> This appears to be a bug. DFDL Spec says:
>
> When the number of "S" symbols in a pattern exceeds the supported accuracy, excess fractional seconds are truncated from the right (not rounded) when parsing, and zeros are added to the right when unparsing. For example, a DFDL processor allows up to six "S" symbols and has millisecond accuracy; for pattern "ss.SSSSSS", data "12.345678" would be parsed into Infoset xs:time "00:00:12:345", which would be unparsed into data "12.345000".
>
> This suggests I should be getting all 6 characters for SSSSSS, even if
> it loses precision of the 000003 I should get 000000.
>
> But I should not be getting just 000.
>
> A bug? Suggestions for how to work around?
>
Re: issue with calendarPattern with SSSSSS not unparsing to 000000
Posted by Steve Lawrence <st...@gmail.com>.
I'm pretty sure we just use ICU4J for unparsing times.
We do have custom code to parse the XML infoset dateTime in
DFDLCalendarConversion.scala. So it's possible we're just parsing that
string from the infoset incorrectly. Skimming that code to see if
there's something obviously wrong, I see this comment in timePartFromString:
// ICU only supports integer milliseconds precision, which means we can
// only support at most 3 digits from the milliseconds field
And in timePartToString I see that we get the subsecond part and pad
three 0's when creating the infoset.
Seems like ICU doesn't support that much precision, and I'd guess it
doesn't padd to the number of S's in the pattern.
Might be another case where ICU behavior and the DFDL specification
don't agree.
On 10/14/20 3:15 PM, Beckerle, Mike wrote:
> I did not find JIRA tickets for this issue. Just want to see if anyone has insight/workaround for this.
>
> This problem occurs for Daffodil 2.4.0 and 2.7.0
>
> I have a schema with
>
> <xs:element
> name="TimestampNano"
> type="xs:dateTime"
> dfdl:calendarPattern="yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZZZZ" />
>
> Notice the 6 S in SSSSSS.
>
> This parses this fine:
>
> 2003-08-24T05:14:15.000003-07:00
>
> producing this XML Infoset:
>
> <TimestampNano>2003-08-24T05:14:15.000003-07:00</TimestampNano>
>
> So it appears that Daffodil supports microseconds. But that unparses to:
>
> 2003-08-24T05:14:15.000-07:00
>
> This appears to be a bug. DFDL Spec says:
>
> When the number of "S" symbols in a pattern exceeds the supported accuracy, excess fractional seconds are truncated from the right (not rounded) when parsing, and zeros are added to the right when unparsing. For example, a DFDL processor allows up to six "S" symbols and has millisecond accuracy; for pattern "ss.SSSSSS", data "12.345678" would be parsed into Infoset xs:time "00:00:12:345", which would be unparsed into data "12.345000".
>
> This suggests I should be getting all 6 characters for SSSSSS, even if
> it loses precision of the 000003 I should get 000000.
>
> But I should not be getting just 000.
>
> A bug? Suggestions for how to work around?
>