You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "gdesrosiers1805 (via GitHub)" <gi...@apache.org> on 2026/06/11 04:37:14 UTC
[PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
gdesrosiers1805 opened a new pull request, #1682:
URL: https://github.com/apache/daffodil/pull/1682
ICU calendars are millisecond-precision, so fractional microseconds in xs:dateTime values are not preserved on parse. Truncate fractional seconds to milliseconds before building XMLGregorianCalendar for comparison, so sub-millisecond differences are ignored rather than flagged as mismatches.
DAFFODIL-3086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3398195058
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
alright, sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397802659
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
I think i copied one of the examples and changed what I needed for these tests, but i agree that makes sense. if you look at this for instance it uses the .dfdl.xsd extension as well.
```<tdml:defineSchema name="DoubleToRawLong.dfdl.xsd">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<dfdl:format ref="ex:GeneralFormat" lengthUnits="bytes" representation="binary" />
<xs:element name="root">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:double" name="DoubleValue" dfdl:lengthKind="explicit" dfdl:length="8" />
<xs:element type="xs:long" name="LongValue" dfdl:inputValueCalc="{ dfdlx:doubleToRawLong(../ex:DoubleValue) }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>
```
I will go ahead and update that one but there might be plenty of other instances we might need to update as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397923112
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
Hmm interesting functions.tdml is pretty old, maybe that's why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397711939
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
Typically I think embedded schemas don't use the .dfdl.xsd schema in the name, this just makes it easier when people are looking at the test to know what's a real file and what's actually embedded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3396857461
##########
daffodil-core/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala:
##########
@@ -1302,6 +1302,16 @@ Differences were (path, expected, actual):
}
}
+ /**
+ * Normalizing fractional seconds to milliseconds precision.
+ * @param data Date/Time
+ * @return normalized data with microseconds precision dropped.
+ */
+ private def normalizeFractionalSeconds(data: String): String = {
+ // match ".ffffff" after seconds, keep at most 3 digits
+ data.replaceAll("""(\.\d{3})\d+""", "$1")
Review Comment:
I actually like that approach better. looks cleaner and less expensive than the regex possibly, although there is likely no noticeable differences here. I also like the normalizeXmlCalendar name suggested below as well. there is already the year0 that needs to be normalized from the other ticket i am also looking at but unfortunately this one has to be preprocessed because the XMLGregorianCalendar does not accept year zero and will throw an IllegalArgumentException on that unlike ICU. it would be nice to have one normalized method for both, but because of the nature of the two cases that is not possible here. Nonetheless I still think normalizing the calendar for the precision makes more sense here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence merged PR #1682:
URL: https://github.com/apache/daffodil/pull/1682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397802659
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
I think i copied one of the examples and changed what I needed for these tests, but i agree that makes sense. if you look at this for instance it uses the .dfdl.xsd extension as well.
`<tdml:defineSchema name="DoubleToRawLong.dfdl.xsd">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<dfdl:format ref="ex:GeneralFormat" lengthUnits="bytes" representation="binary" />
<xs:element name="root">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:double" name="DoubleValue" dfdl:lengthKind="explicit" dfdl:length="8" />
<xs:element type="xs:long" name="LongValue" dfdl:inputValueCalc="{ dfdlx:doubleToRawLong(../ex:DoubleValue) }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>`
I will go ahead and update that one but there might be plenty of other instances we might need to update as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3395533180
##########
daffodil-core/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala:
##########
@@ -1302,6 +1302,16 @@ Differences were (path, expected, actual):
}
}
+ /**
+ * Normalizing fractional seconds to milliseconds precision.
+ * @param data Date/Time
+ * @return normalized data with microseconds precision dropped.
+ */
+ private def normalizeFractionalSeconds(data: String): String = {
Review Comment:
Thoughts on calling this something like `normalizeXmlCalendar`, and then if we ever need additional normalizations we can put them all in here and the name still makes sense?
##########
daffodil-core/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala:
##########
@@ -1302,6 +1302,16 @@ Differences were (path, expected, actual):
}
}
+ /**
+ * Normalizing fractional seconds to milliseconds precision.
+ * @param data Date/Time
+ * @return normalized data with microseconds precision dropped.
+ */
+ private def normalizeFractionalSeconds(data: String): String = {
+ // match ".ffffff" after seconds, keep at most 3 digits
+ data.replaceAll("""(\.\d{3})\d+""", "$1")
Review Comment:
Thoughts on normalizing the XMLGregorianCalendar after it's parsed, instead of pre-processing the string? I can't think of a case where this regex will do the wrong thing, but it feels like it woudl be safer and more clear to do it after being parsed, e.g.:
```scala
private def normalizeFractionalSeconds(cal: XMLGregorianCalendar): String = {
val frac = cal.getFractionalSeconds()
if (frac != null) {
cal.setFractionalSeconds(frac.setScale(3, RoundingMode.DOWN))
}
}
```
It's a bit more verbose, but it feels a bit more clear that it's normalizing the fractional seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397802659
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
I think i copied one of the examples and changed what I needed for these tests, but i agree that makes sense. if you look at this for instance it uses the .dfdl.xsd extension as well.
<tdml:defineSchema name="DoubleToRawLong.dfdl.xsd">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<dfdl:format ref="ex:GeneralFormat" lengthUnits="bytes" representation="binary" />
<xs:element name="root">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:double" name="DoubleValue" dfdl:lengthKind="explicit" dfdl:length="8" />
<xs:element type="xs:long" name="LongValue" dfdl:inputValueCalc="{ dfdlx:doubleToRawLong(../ex:DoubleValue) }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>
I will go ahead and update that one but there might be plenty of other instances we might need to update as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397802659
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
I think i copied one of the examples and changed what I needed for these tests, but i agree that makes sense. if you look at this for instance it uses the .dfdl.xsd extension as well.
```<tdml:defineSchema name="DoubleToRawLong.dfdl.xsd">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<dfdl:format ref="ex:GeneralFormat" lengthUnits="bytes" representation="binary" />
<xs:element name="root">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:double" name="DoubleValue" dfdl:lengthKind="explicit" dfdl:length="8" />
<xs:element type="xs:long" name="LongValue" dfdl:inputValueCalc="{ dfdlx:doubleToRawLong(../ex:DoubleValue) }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>```
I will go ahead and update that one but there might be plenty of other instances we might need to update as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3398032060
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
there is only a few of those in that file that have the extension so I could update them all there, even though it is really unrelated to this PR. I am fine with whatever we think is best whether doing it here or as a separate PR. @stevedlawrence do you have any preference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397857839
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
looking at functions.tdml we are pretty inconsistent between the two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397256316
##########
daffodil-core/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala:
##########
@@ -1302,6 +1302,16 @@ Differences were (path, expected, actual):
}
}
+ /**
+ * Normalizing fractional seconds to milliseconds precision.
+ * @param data Date/Time
+ * @return normalized data with microseconds precision dropped.
+ */
+ private def normalizeFractionalSeconds(data: String): String = {
+ // match ".ffffff" after seconds, keep at most 3 digits
+ data.replaceAll("""(\.\d{3})\d+""", "$1")
Review Comment:
I think it's fine if year zero throws IllegalArgumentException. That is not valid XSD so Daffodil should never create it (once we fix the referenced issue). If a TDML file has year 0 as an expected infoset then it's fine for that test to fail with a TDML exception about an invalid date/time field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "gdesrosiers1805 (via GitHub)" <gi...@apache.org>.
gdesrosiers1805 commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3397802659
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
I think i copied one of the examples and changed what I needed for these tests, but i agree that makes sense. if you look at this for instance it uses the .dfdl.xsd extension as well.
```
<tdml:defineSchema name="DoubleToRawLong.dfdl.xsd">
<xs:include schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<dfdl:format ref="ex:GeneralFormat" lengthUnits="bytes" representation="binary" />
<xs:element name="root">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:double" name="DoubleValue" dfdl:lengthKind="explicit" dfdl:length="8" />
<xs:element type="xs:long" name="LongValue" dfdl:inputValueCalc="{ dfdlx:doubleToRawLong(../ex:DoubleValue) }" />
</xs:sequence>
</xs:complexType>
</xs:element>
</tdml:defineSchema>
```
I will go ahead and update that one but there might be plenty of other instances we might need to update as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Tolerate sub-millisecond differences in TDML dateTime comparison [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1682:
URL: https://github.com/apache/daffodil/pull/1682#discussion_r3398156173
##########
daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml:
##########
@@ -14735,7 +14735,94 @@
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
-
+
+
+ <tdml:defineSchema name="MicrosecondsDateTime.dfdl.xsd">
Review Comment:
Agreed with Lola. It looks like the convention is mostly to not use .dfdl.xsd for embedded schemas. We have 600+ defineSchemas that do not use it and 80+ that do.
So I would suggest drop the .dfdl.xsd for new tests, but keep the old ones as is. I'm not sure it's worth teh code churn to updating our existing tests for something that is only a loose convention. If we have a good argument for it, we can do it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org