You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by GitBox <gi...@apache.org> on 2020/07/10 01:06:35 UTC
[GitHub] [incubator-daffodil] mbeckerle opened a new pull request #395: Add test_nulPattern1
mbeckerle opened a new pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395
Shows that <xs:pattern value="*"/> doesn't work as a
pattern to match the E000 substitution characters that Daffodil
uses in XML to replace ASCII NUL characters.
This just adds the test showing the problem. Doesn't attempt to fix it.
DAFFODIL-2363
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452865746
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
So using both E000 and 0000 won't work because you can't say � in an XML document of any kind. We would have to interpret the pattern looking for DFDL character entities and allow [%#x0000;]* for DFDL's internal validation, but then we'd have to remove the %#x0000; for Xerces XML Validation.
Our internal validation needs to take the pattern, and apply the same backward transformation to its characters that we use for XML data. So the internal validation would convert that  back into a true NUL character that it is searching for in the infoset. That appears to be the bug.
Another alternative, I wasn't aware that when we turn Validation On, that *both* our internal validation AND Xerces run. Maybe we should have a Validation="xml" setting that only runs Xerces validation. This is also perhaps a special DFDL-language extension mode, where you can put any XML Schema validations at all, including key-constraints, unique constraints, patterns on hexBinary, all the things DFDL usually disallows, because the validations would be used ONLY for XML validation of an XML infoset. I suppose a few constraints on what you could write for facets would still apply, such as things about single-byte charsets when lengthUnits is bytes and lengthKind is "implicit" for strings.
There is another ticket for lifting DFDL's restrictions on key/unique constraints, again so that people can write one schema and use it for DFDL parse/unparse, as well as for more comprehensive XML Schema validation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] tuxji commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
tuxji commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452832879
##########
File path: daffodil-test/src/test/scala/org/apache/daffodil/section05/facets/TestNulChars.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.section05.facets
+
+import org.apache.daffodil.tdml.Runner
+import org.junit.AfterClass
+import org.junit.Test
+
+object TestNulChars {
+ lazy val runner = Runner("/org/apache/daffodil/section05/facets", "NulChars.tdml")
+
+ @AfterClass def shutDown = {
+ runner.reset
+ }
+}
+
+import TestNulChars._
+
+class TestNulChars {
+
+ // DAFFODIL-2363  (NUL replacement into XML) can't be used in pattern facet. With full validation.
+ // @Test def test_nulPattern1() = { runner.runOneTest("nulPattern1") }
+
+ // DAFFODIL-2364 - infinite loop
+ // @Test def test_nulPad1() = { runner.runOneTest("nulPad1") }
+
+ @Test def ignoreThisDummyTestToPreventIDEFromRemovingImports() = {
+ // TODO Delete this dummy test once at least one of the above tests is made to work.
+ runner.asInstanceOf[AnyRef]
Review comment:
This dummy test actually fails too:
`[error] Test org.apache.daffodil.section05.facets.TestNulChars.initializationError failed: java.lang.Exception: Method ignoreThisDummyTestToPreventIDEFromRemovingImports() should be void, took 0.002 sec`
I suggest changing line 41 to:
` @Test def ignoreThisDummyTestToPreventIDEFromRemovingImports(): Unit = {`
I also grep'ed @Test and noticed the rest of the tests are inconsistent; the majority of them have the explicit : Unit type, but some omit it (they have just () = { ...).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle merged pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle merged pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452865746
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
So using both E000 and 0000 won't work because you can't say &#x0000; in an XML document of any kind. We would have to interpret the pattern looking for DFDL character entities and allow [&#xE000;%#x0000;]* for DFDL's internal validation, but then we'd have to remove the %#x0000; for Xerces XML Validation, but that could leave you with an inconsistent regex.
Our internal validation needs to take the pattern, and apply the same backward transformation to its characters that we use for XML data. So the internal validation would convert that  back into a true NUL character that it is searching for in the infoset. That appears to be the bug.
Another alternative, I wasn't aware that when we turn Validation On, that *both* our internal validation AND Xerces run. Maybe we should have a Validation="xml" setting that only runs Xerces validation. This is also perhaps a special DFDL-language extension mode, where you can put any XML Schema validations at all, including key-constraints, unique constraints, patterns on hexBinary, all the things DFDL usually disallows, because the validations would be used ONLY for XML validation of an XML infoset. I suppose a few constraints on what you could write for facets would still apply, such as things about single-byte charsets when lengthUnits is bytes and lengthKind is "implicit" for strings.
There is another ticket for lifting DFDL's restrictions on key/unique constraints, again so that people can write one schema and use it for DFDL parse/unparse, as well as for more comprehensive XML Schema validation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] tuxji commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
tuxji commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452849693
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
I read [DAFFODIL-2363](https://issues.apache.org/jira/browse/DAFFODIL-2363) for more context. If the goal is to capture NUL padding regions and check they are all NUL characters, then Steve's suggestion to change the pattern sounds like it might work. Would that be good enough (just change the pattern) or do we need to check what the [DFDL specification](http://daffodil.apache.org/docs/dfdl/) says about how the validation should be performed in this edge case? This section may apply:
> 5.2.4 Pattern
> · Allowed only on elements of type xs:string or types derived from it in Figure 3 DFDL simple types.
> · Used for validation only
> It is important to avoid confusion of the pattern facet with other uses of regular expressions that are needed in DFDL (for example, to determine the length of an element by regular expression matching).
>Note: in XSD, pattern is about the lexical representation of the data, and since all is text there, everything has a lexical representation. In DFDL only strings are guaranteed to have a lexical and logical value that is identical.
Also, should the pattern represent a NUL (character 0) codepoint using a DFDL character entity?
> This NUL character code is not allowed in XML documents, including DFDL Schemas; hence, it must be specified using a DFDL Character Entity. Within a DFDL Expression, use the dfdl:decodeDFDLEntities() function to obtain a string containing this character.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452866839
##########
File path: daffodil-test/src/test/scala/org/apache/daffodil/section05/facets/TestNulChars.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.section05.facets
+
+import org.apache.daffodil.tdml.Runner
+import org.junit.AfterClass
+import org.junit.Test
+
+object TestNulChars {
+ lazy val runner = Runner("/org/apache/daffodil/section05/facets", "NulChars.tdml")
+
+ @AfterClass def shutDown = {
+ runner.reset
+ }
+}
+
+import TestNulChars._
+
+class TestNulChars {
+
+ // DAFFODIL-2363  (NUL replacement into XML) can't be used in pattern facet. With full validation.
+ // @Test def test_nulPattern1() = { runner.runOneTest("nulPattern1") }
+
+ // DAFFODIL-2364 - infinite loop
+ // @Test def test_nulPad1() = { runner.runOneTest("nulPad1") }
+
+ @Test def ignoreThisDummyTestToPreventIDEFromRemovingImports() = {
+ // TODO Delete this dummy test once at least one of the above tests is made to work.
+ runner.asInstanceOf[AnyRef]
Review comment:
Grrr. We really should put in those ": Unit" types, if the compiler is going to check this, which it should for normal coding. It only seems like clutter because these test driver things are one-liners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452865746
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
So using both E000 and 0000 won't work because you can't say &#x0000; in an XML document of any kind. We would have to interpret the pattern looking for DFDL character entities and allow [&#xE000;%#x0000;]* for DFDL's internal validation, but then we'd have to remove the %#x0000; for Xerces XML Validation, but that could leave you with an inconsistent regex.
Our internal validation needs to take the pattern, and apply the same backward transformation to its characters that we use for XML data. So the internal validation would convert that &#xE000; back into a true NUL character that it is searching for in the infoset. That appears to be the bug.
Another alternative, I wasn't aware that when we turn Validation On, that *both* our internal validation AND Xerces run. Maybe we should have a Validation="xml" setting that only runs Xerces validation. This is also perhaps a special DFDL-language extension mode, where you can put any XML Schema validations at all, including key-constraints, unique constraints, patterns on hexBinary, all the things DFDL usually disallows, because the validations would be used ONLY for XML validation of an XML infoset. I suppose a few constraints on what you could write for facets would still apply, such as things about single-byte charsets when lengthUnits is bytes and lengthKind is "implicit" for strings.
There is another ticket for lifting DFDL's restrictions on key/unique constraints, again so that people can write one schema and use it for DFDL parse/unparse, as well as for more comprehensive XML Schema validation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452788392
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
I suspect the issue here is that the infoset and XML actually have different values during different times of validation.
Our internal infoset representation has actual NUL characters in them. I think this is correct since some infoset representations (like json) have no problem handling NUL characters. This means the the xE000 pattern will fail when Daffodil performs it's validation.
When we perform Xerces validation, we generate XML and thus convert those NUL characters to 0xE000. Which will pass the Xerces check.
So the infoset looks different to Daffodil and Xerces, but are using the same pattern. It seems like this can never work unless you change the pattern to something like ($#x0000*|$#xE000), or if we change how we handle XML illegal characters. Or should Daffodil validation do an XML conversion and then validate? I'm not sure what the right behavior is, perhaps this is expected behavior and not a bug?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #395: Add test_nulPattern1
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #395:
URL: https://github.com/apache/incubator-daffodil/pull/395#discussion_r452865746
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section05/facets/NulChars.tdml
##########
@@ -0,0 +1,95 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<tdml:testSuite
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:ex="http://example.com"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ defaultRoundTrip="onePass"
+ defaultValidation="on">
+
+ <tdml:defineSchema name="nullTest">
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"/>
+
+ <xs:element name="root">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="foo" dfdl:lengthKind="pattern" dfdl:lengthPattern="\x{0000}{3,3}">
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:pattern value="*"/>
Review comment:
So using both E000 and 0000 won't work because you can't say &#x0000; in an XML document of any kind. We would have to interpret the pattern looking for DFDL character entities and allow [%#x0000;]* for DFDL's internal validation, but then we'd have to remove the %#x0000; for Xerces XML Validation.
Our internal validation needs to take the pattern, and apply the same backward transformation to its characters that we use for XML data. So the internal validation would convert that  back into a true NUL character that it is searching for in the infoset. That appears to be the bug.
Another alternative, I wasn't aware that when we turn Validation On, that *both* our internal validation AND Xerces run. Maybe we should have a Validation="xml" setting that only runs Xerces validation. This is also perhaps a special DFDL-language extension mode, where you can put any XML Schema validations at all, including key-constraints, unique constraints, patterns on hexBinary, all the things DFDL usually disallows, because the validations would be used ONLY for XML validation of an XML infoset. I suppose a few constraints on what you could write for facets would still apply, such as things about single-byte charsets when lengthUnits is bytes and lengthKind is "implicit" for strings.
There is another ticket for lifting DFDL's restrictions on key/unique constraints, again so that people can write one schema and use it for DFDL parse/unparse, as well as for more comprehensive XML Schema validation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org