You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by GitBox <gi...@apache.org> on 2020/10/21 17:35:07 UTC
[GitHub] [incubator-daffodil] mbeckerle opened a new pull request #442: Enable pattern facet to use PUA characters
mbeckerle opened a new pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442
Enables patterns to validate the DFDL infoset
characters that are otherwise XML-Illegal such as
0xB or NUL (0x0), by expressing them as  or

Test test_nulPattern1 verifies this.
DAFFODIL-2363
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] olabusayoT commented on a change in pull request #442: Enable pattern facet to use PUA characters
Posted by GitBox <gi...@apache.org>.
olabusayoT commented on a change in pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442#discussion_r509519530
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Facets.scala
##########
@@ -18,9 +18,11 @@
package org.apache.daffodil.dsom
import java.math.BigInteger
+
Review comment:
IntelliJ separates the java imports by default. So if it did anything with imports, it might have formatted them
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle merged pull request #442: Enable pattern facet to use PUA characters
Posted by GitBox <gi...@apache.org>.
mbeckerle merged pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #442: Enable pattern facet to use PUA characters
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442#discussion_r509564352
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Facets.scala
##########
@@ -129,10 +131,45 @@ trait Facets { self: Restriction =>
final lazy val patternValues: Seq[FacetValueR] = {
val values = combinedBaseFacets.filter { case (f, _) => f == Facet.pattern }
if (values.size > 0) {
- val res: Seq[FacetValueR] = values.map { case (f, v) => (f, v.r) }
+ val res: Seq[FacetValueR] = values.map {
+ case (f, v) => {
+ //
+ // The DFDL Infoset can contain strings which hold characters
+ // that are not allowed in XML at all.
+ //
+ // In order to talk about these characters in a XSD pattern facet
+ // we use a remapping of such characters into the Unicode
+ // Private Use Area, so as to have XML-legal characters.
+ //
+ // See the section titled "XML Illegal Characters" on this web page:
+ // https://daffodil.apache.org/infoset/
+ //
+ // Before processing a regex of these characters in Daffodil's pattern facet
+ // validation, we must remap these PUA characters back to the originally
+ // intended code points, since that's what the Infoset strings will contain.
+ //
+ // Consider the character code 0xB. This is illegal in XML v1.0 documents.
+ // A DFDL Schema is an XML Schema, which is an XML document; hence, we cannot
+ // use the character with code 0xB directly, nor can we use an XML numeric
+ // character entity like  for it. The character is simply disallowed in
+ // XML, including DFDL schemas. Hence, we mention 0xB by using a remapping of it
+ // to the PUA area character 0XE00B, which we express by 
+ // Hence a pattern regex like "[ 0-9a-zA-Z]" will match
+ // the character with char code 0xB (remapped from E00B), as well as spaces,
+ // and alphanumeric characters.
Review comment:
Yeah, it's a bit odd, but in any XML context, including the pattern regex, if you want to talk about an XML-illegal character you have to use the PUA character instead. A DFDL schema is an XML document, so the pattern facet regex has to do this also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #442: Enable pattern facet to use PUA characters
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442#discussion_r509485044
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Facets.scala
##########
@@ -18,9 +18,11 @@
package org.apache.daffodil.dsom
import java.math.BigInteger
+
Review comment:
Remove blank line. Not sure at all how this got inserted.
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Facets.scala
##########
@@ -129,10 +131,45 @@ trait Facets { self: Restriction =>
final lazy val patternValues: Seq[FacetValueR] = {
val values = combinedBaseFacets.filter { case (f, _) => f == Facet.pattern }
if (values.size > 0) {
- val res: Seq[FacetValueR] = values.map { case (f, v) => (f, v.r) }
+ val res: Seq[FacetValueR] = values.map {
+ case (f, v) => {
+ //
+ // The DFDL Infoset can contain strings which hold characters
+ // that are not allowed in XML at all.
+ //
+ // In order to talk about these characters in a XSD pattern facet
+ // we use a remapping of such characters into the Unicode
+ // Private Use Area, so as to have XML-legal characters.
+ //
+ // See the section titled "XML Illegal Characters" on this web page:
+ // https://daffodil.apache.org/infoset/
+ //
+ // Before processing a regex of these characters in Daffodil's pattern facet
+ // validation, we must remap these PUA characters back to the originally
+ // intended code points, since that's what the Infoset strings will contain.
+ //
+ // Consider the character code 0xB. This is illegal in XML v1.0 documents.
+ // A DFDL Schema is an XML Schema, which is an XML document; hence, we cannot
+ // use the character with code 0xB directly, nor can we use an XML numeric
+ // character entity like  for it. The character is simply disallowed in
+ // XML, including DFDL schemas. Hence, we mention 0xB by using a remapping of it
+ // to the PUA area character 0XE00B, which we express by 
Review comment:
Lower case "x"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #442: Enable pattern facet to use PUA characters
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #442:
URL: https://github.com/apache/incubator-daffodil/pull/442#discussion_r509548706
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/Facets.scala
##########
@@ -129,10 +131,45 @@ trait Facets { self: Restriction =>
final lazy val patternValues: Seq[FacetValueR] = {
val values = combinedBaseFacets.filter { case (f, _) => f == Facet.pattern }
if (values.size > 0) {
- val res: Seq[FacetValueR] = values.map { case (f, v) => (f, v.r) }
+ val res: Seq[FacetValueR] = values.map {
+ case (f, v) => {
+ //
+ // The DFDL Infoset can contain strings which hold characters
+ // that are not allowed in XML at all.
+ //
+ // In order to talk about these characters in a XSD pattern facet
+ // we use a remapping of such characters into the Unicode
+ // Private Use Area, so as to have XML-legal characters.
+ //
+ // See the section titled "XML Illegal Characters" on this web page:
+ // https://daffodil.apache.org/infoset/
+ //
+ // Before processing a regex of these characters in Daffodil's pattern facet
+ // validation, we must remap these PUA characters back to the originally
+ // intended code points, since that's what the Infoset strings will contain.
+ //
+ // Consider the character code 0xB. This is illegal in XML v1.0 documents.
+ // A DFDL Schema is an XML Schema, which is an XML document; hence, we cannot
+ // use the character with code 0xB directly, nor can we use an XML numeric
+ // character entity like  for it. The character is simply disallowed in
+ // XML, including DFDL schemas. Hence, we mention 0xB by using a remapping of it
+ // to the PUA area character 0XE00B, which we express by 
+ // Hence a pattern regex like "[ 0-9a-zA-Z]" will match
+ // the character with char code 0xB (remapped from E00B), as well as spaces,
+ // and alphanumeric characters.
Review comment:
Just so I'm clear, ```` is disallowed in an XSD pattern regex because XSD is just a special XML. And based on Daffodil's PUA handling, one must instead use ``&xE00B;``? Then this transforms that into a regex that matches 0x0B? And our internal infoset value that we check this pattern against hasn't been mapped to PUA at the time we run this regex, because we only transform infoset values to PUA right before converting to XML with an XML infoset outputter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org