You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by GitBox <gi...@apache.org> on 2020/11/25 01:45:49 UTC
[GitHub] [incubator-daffodil] jw3 opened a new pull request #463: Embedded Schematron
jw3 opened a new pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463
Enable validation using schematron embedded in dfdl xsd.
DAFFODIL-
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549490284
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
I cant yet generate a failing test even after enabling xslt 2.0.
> E.g., you cannot just type string-length(../some/path) in DFDL nor XPath 2.0 and expect it to work. You must bind a prefix (usually "fn" to namespace http://www.w3.org/2005/xpath-functions/ and use the prefix when calling the functions.
In context of xpath, are you saying "must" because there is enforcement or because its good practice?
Perhaps a counter example example is the GIF schema that uses unprefixed functions and enables xslt 2.0.
With all that said, I think the tools are all there, the included version of schematron supports either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549490284
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
I cant yet generate a failing test even after enabling xslt 2.0.
> E.g., you cannot just type string-length(../some/path) in DFDL nor XPath 2.0 and expect it to work. You must bind a prefix (usually "fn" to namespace http://www.w3.org/2005/xpath-functions/ and use the prefix when calling the functions.
In context of xpath, are you saying "must" because there is enforcement or because its good practice?
Perhaps a counter example example is the GIF schema that uses unprefixed functions and enables xslt 2.0.
https://github.com/DFDLSchemas/GIF/blob/master/src/main/resources/com/mitre/gif/sch/GIF.sch
With all that said, I think the tools are all there, the included version of schematron supports either way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r537575090
##########
File path: daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
##########
@@ -0,0 +1,107 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ based on an original transform by Eddie Robertsson
+ 2001/04/21 fn: added support for included schemas
+ 2001/06/27 er: changed XMl Schema prefix from xsd: to xs: and changed to the Rec namespace
+ 2010/04/14 rj: Update for ISO Schematron using xslt2
+-->
+
Review comment:
Remember to add this new file to the three LICENSE files in LICENSE, daffodil-cli/LICENSE, and in daffodil-schematron/src/main/resources/....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-762847407
:+1: Looks really nice! Loving the recent improvements to the CLI tests, hoping that can be the basis for cleaning up all of our CLI test boilerplate in the future. Just need to get things passing on windows, and squash all the commits and I think this ready to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-748959533
@rthomas320
> If SVRL reports were generated would it be possible to commit those to git also?
I can post the SVRL generated by a unit test or two here in the PR.
(replied to the other comments on the dev list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] create jira ticket
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] update LICENSE and NOTICE files appropriately
- [ ] tests with complex schematron
- [ ] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-757943291
+1 Looks nice! Just a couple minor comments about the licenses and maybe add some CLI testing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r542573243
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -30,13 +30,19 @@
<sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
</sch:rule>
</sch:pattern>
+ <sch:pattern name="test-short-uuid">
+ <sch:rule context="short-uuid">
+ <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
Review comment:
> I was expecting to see this rule as an annotation on the element declaration of short-uuid below.
The first pass had it embedded and it looked like it was going to get noisy.
I can see advantages of both.. will add an example for this.
> W.r.t this test, I was hoping for a test using a sch rule that would use some relative paths like ../../foo/bar to reach around in the infoset and constrain disparate things together.
I see. Will add examples.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-748959533
@rthomas320
> If SVRL reports were generated would it be possible to commit those to git also?
I can post the SVRL generated by a unit test or two here in the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560172664
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section02/schema_definition_errors/SchemaDefinitionErrors.tdml
##########
@@ -203,9 +203,11 @@
description="">
<tdml:document><![CDATA[test]]></tdml:document>
- <tdml:errors>
- <tdml:error />
- </tdml:errors>
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <elem>test</elem>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
Review comment:
Ah, ok. Handling the diagnostics output on stderr in the cli tests must have had me thinking this was checked, but I see what you mean now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r548263498
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
All the references I see say there is a full set of operators in both xpath 1 and 2. Here is one of several without looking too hard https://www.w3schools.com/xml/xpath_operators.asp
Curious as to what I am missing in this discussion about limited xpath operators?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r547998872
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
Thanks. What about the namespaces for the function names?
The reason I focus on this is that DFDL expressions are XPath 2.0 derived, though not identically those, but DFDL expressions the function names are in a namespace, which usually requires a namespace prefix like "fn:".
This inconsistency is manageable, but unfortunate if expressions like these that could, in principle be either DFDL expressions or schematron expressions, can't be cut/pasted between schematron rules and say, DFDL asserts, because the expression languages are that different.
This really only matters for very simple expressions. DFDL expressions basically can't do anything that can't execute in constant time. If it involves searching, counting, or really any sort of iteration over the infoset tree, then DFDL expressions don't allow it for the most part, and Daffodil doesn't implement it. Those are the kinds of things that you need schematron for. So DFDL expressions can already check lengths, but a query like "all FOO elements must be beneath a BAR element" or "there must be 3 or more FOO children of each QUUX element" can't be expressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555074093
##########
File path: daffodil-cli/bin.LICENSE
##########
@@ -876,3 +876,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
Review comment:
Yep. Looks like those have the same issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-764838075
Looks good! I think you've got all the requisite +1's. Feel free to merge and resolve the Jira ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-743235168
### reference
- https://reference.niem.gov/niem/specification/sch-in-xsd/1.0alpha4/niem-schematron-in-xsd-1.0alpha4-2016-03-16.html
- https://journal.code4lib.org/articles/13546
- https://www.mulberrytech.com/papers/schematron-Philly.pdf
- http://schematronist.org/mailman/listinfo/schematron_schematronist.org
(some of these might make the readme, might not, here for posterity)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555064689
##########
File path: daffodil-schematron/src/test/resources/xsd/always-fails-1.dfdl.xsd
##########
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ </xs:annotation>
+
+ <xs:element name="always-fails" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator="">
+ <xs:annotation>
+ <xs:appinfo>
Review comment:
Importing the ns is the only requirement I have saw. In this case schematron is imported from the top level schema tag, there are a couple tests that import from an appinfo. Otherwise nothing noteworthy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-751927910
Squashed and rebased
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560159726
##########
File path: daffodil-test/src/test/resources/org/apache/daffodil/section02/schema_definition_errors/SchemaDefinitionErrors.tdml
##########
@@ -203,9 +203,11 @@
description="">
<tdml:document><![CDATA[test]]></tdml:document>
- <tdml:errors>
- <tdml:error />
- </tdml:errors>
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <elem>test</elem>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
Review comment:
Just want to point out that if there is not a ``<tdml:warning>`` element in a test, the TDML runner doesn't care if there are warnings or not. The TMDL Runner only requires that warnings exist if the ``<tdml:warnigns>`` element exists. So unfortunately there's no way test that no warnings were output. Probably still worth having this, since some point we will want to add that feature and if we are ignoring a warning here, that would be caught some day.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549485709
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
Setting `queryBinding="xslt2"` on the root element will enable xslt+xpath 2.0 with our current version of schematron.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-743235168
### reference
- https://reference.niem.gov/niem/specification/sch-in-xsd/1.0alpha4/niem-schematron-in-xsd-1.0alpha4-2016-03-16.html
- https://journal.code4lib.org/articles/13546
- https://www.mulberrytech.com/papers/schematron-Philly.pdf
-
(some of these might make the readme, might not, here for posterity)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] create jira ticket
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] update LICENSE and NOTICE files appropriately
- [ ] tests with non-trivial schematron
- [x] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] create jira ticket
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] update LICENSE and NOTICE files appropriately
- [ ] tests with complex schematron
- [x] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555052331
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/EmbeddedTesting.scala
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.validation.schematron
+
+import org.apache.daffodil.sapi.Daffodil
+import org.apache.daffodil.sapi.DataProcessor
+import org.apache.daffodil.sapi.Diagnostic
+import org.apache.daffodil.sapi.ParseResult
+import org.apache.daffodil.sapi.infoset.XMLTextInfosetOutputter
+import org.apache.daffodil.sapi.io.InputSourceDataInputStream
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.validation.schematron.SchSource.Xsd
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.File
+
+trait EmbeddedTesting {
+ def withJpgSchema(f: Validation => Unit): Unit = withSchema("xsd/jpeg.dfdl.xsd")(f)
+ def withBmpSchema(f: Validation => Unit): Unit = withSchema("xsd/bmp.dfdl.xsd")(f)
+ def withGifSchema(f: Validation => Unit): Unit = withSchema("xsd/gif.dfdl.xsd")(f)
Review comment:
Yep, good catch. Those schemas got copied into here to help a while and were deleted, missed this bit of cleanup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733925743
A good thing to do with this (and the prior) PR that provide schematron, is to fix the JPEG schema that is published on github to use this new feature. Version 1 would incorporate that JPEG schema's existing ".sch" schematron rules file. Version 2 would migrate those rules up into the DFDL schema itself, where IMHO, they belong.
Without the schematron validation, that DFDL JPEG schema is known to be very poor at rejecting bad data. It's the specific example where we learned it is very important to make DFDL schemas not only parse correct data, but reject invalid and/or malformed data. Without the schematron rules, this JPEG schema accepts all sorts of malformed stuff. This more powerful validator in this case enforces, not only validity, but well-formedness even.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555067788
##########
File path: daffodil-cli/bin.LICENSE
##########
@@ -876,3 +876,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
Review comment:
ok. looks like the other schematron files above need corrected also, agree?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549551638
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/TestBasicValidation.scala
##########
@@ -68,7 +68,7 @@ class TestBasicValidation {
val factory = new TransformerFactoryImpl()
factory.setURIResolver(resolver)
val sch = getClass.getClassLoader.getResourceAsStream("sch/schematron-1.sch")
- val rules = Transforms.fromSch(sch, factory)
+ val rules = Transforms.from(sch, Sch, factory)
Review comment:
added DAFFODIL-2448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r551317055
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -24,25 +24,26 @@
<dfdl:format ref="GeneralFormat"/>
</xs:appinfo>
<xs:appinfo xmlns:sch="http://purl.oclc.org/dsdl/schematron">
- <sch:pattern name="test-uuid">
+ <sch:pattern name="validate-uuid">
<sch:rule context="uuid">
- <sch:assert test="string-length(.)=36">uuid must be 36 chars</sch:assert>
- <sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
- </sch:rule>
- </sch:pattern>
- <sch:pattern name="test-short-uuid">
- <sch:rule context="short-uuid">
- <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
+ <sch:assert test="string-length(time_low)=8"><sch:name/> wrong length</sch:assert>
Review comment:
from [iso_schematron_skeleton_for_saxon.xsl#L66-L70](https://github.com/Schematron/schematron/blob/master/trunk/schematron/code/iso_schematron_skeleton_for_saxon.xsl#L66-L70
)
```
Another tip is that Schematron provides an
element <iso:ns> for declaring the namespaces and prefixes used in Xpaths in
attribute values; it does not extend the XML Namespaces mechanism: if a name
in an XPath has a prefix, there must be an <iso:ns> element for that prefix; if
a name in an XPath does not have a prefix, it is always in no namespace.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560170275
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("2;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("0;0")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+}
Review comment:
Lot's of \n's in this file. Likely going to cause issues on windows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560188221
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
Review comment:
that made for a nice refinement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-764838075
Looks good! I think you've got all the requisite +1's. Feel free to merge and resolve the Jira ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r548263498
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
The references below say there is a full set of operators in xpath
- https://www.w3schools.com/xml/xpath_operators.asp
- https://en.wikipedia.org/wiki/XPath#Functions_and_operators
- https://en.wikipedia.org/wiki/XPath_2.0#Other_operators
Am I missing something here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555024283
##########
File path: LICENSE
##########
@@ -385,3 +385,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
+ The content is available under the MIT License:
+
+ Copyright (c) 2002-2010 Rick Jelliffe and Topologi Pty. Ltd.
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE.
Review comment:
Very minor, but we've been indenting the license 2 more spaces than the "This product bundels..." stuff so it's more clear where the license content is. Same with where the other license files.
##########
File path: daffodil-schematron/src/main/scala/org/apache/daffodil/validation/schematron/ClassPathUriResolver.scala
##########
@@ -31,9 +31,15 @@ final class ClassPathUriResolver(rulesDir: String, fallback: Option[URIResolver]
Option(getClass.getClassLoader.getResourceAsStream(path)) match {
case Some(is) => new StreamSource(is)
case None =>
- fallback.map(_.resolve(href, base)).getOrElse(
- throw ValidatorInitializationException(s"schematron resource not found at $path")
- )
+ // fallback #1;; try the raw classpath without the prefix
+ Option(getClass.getClassLoader.getResourceAsStream(href)) match {
Review comment:
Just curious, what's the case were we need to resolve without the template dir? Is this because a DFDL schema might xs:include/xs:import other schemas that might have embedded schematron annotations?
##########
File path: daffodil-cli/bin.LICENSE
##########
@@ -876,3 +876,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
Review comment:
For the CLI license, the path to this file is actually slightly different and inside a jar. So this should be something like this:
```
iso-schematron-xslt2/ExtractSchFromXSD-2.xsl in lib/org.apache.daffodil.daffodil-schematron-<VERSION>.jar
```
See the W3C license part near the top of the license file for an example
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/EmbeddedTesting.scala
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.validation.schematron
+
+import org.apache.daffodil.sapi.Daffodil
+import org.apache.daffodil.sapi.DataProcessor
+import org.apache.daffodil.sapi.Diagnostic
+import org.apache.daffodil.sapi.ParseResult
+import org.apache.daffodil.sapi.infoset.XMLTextInfosetOutputter
+import org.apache.daffodil.sapi.io.InputSourceDataInputStream
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.validation.schematron.SchSource.Xsd
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.File
+
+trait EmbeddedTesting {
+ def withJpgSchema(f: Validation => Unit): Unit = withSchema("xsd/jpeg.dfdl.xsd")(f)
+ def withBmpSchema(f: Validation => Unit): Unit = withSchema("xsd/bmp.dfdl.xsd")(f)
+ def withGifSchema(f: Validation => Unit): Unit = withSchema("xsd/gif.dfdl.xsd")(f)
Review comment:
We really shouldn't rely on external schemas for testing. I'd prefer we just remove this and ensure our included test files provide sufficient test coverage. Though I assume this means embedded schematron does work with work with those schemas, which is good to hear.
##########
File path: daffodil-schematron/src/main/resources/META-INF/LICENSE
##########
@@ -237,3 +237,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
Review comment:
Like the above comment, this file has a different path since it's inside the daffodil-schematron jar. Unlike the CLI license, you don't need to include the jar since this LICENSE file is already in the jar. So this should just be:
```
iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
```
##########
File path: daffodil-schematron/src/test/resources/xsd/always-fails-1.dfdl.xsd
##########
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ </xs:annotation>
+
+ <xs:element name="always-fails" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator="">
+ <xs:annotation>
+ <xs:appinfo>
Review comment:
Interesting, so embdded schematron rules don't require a special appinfo like DFDL does? I guess the XSLT that extracts these just looks for any any elements that have the schematron namespace?
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/TestEmbeddedSchematron.scala
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.validation.schematron
+
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbeddedSchematron extends EmbeddedTesting {
+ @Test def variation1(): Unit = withSchema("xsd/embedded-1.dfdl.xsd") { f =>
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def variation2(): Unit = withSchema("xsd/embedded-2.dfdl.xsd") { f =>
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def variation3(): Unit = withSchema("xsd/embedded-3.dfdl.xsd") { f =>
+ f.parse(UUID.randomUUID.toString).diagnostics.foreach(println)
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ f.parse(UUID.randomUUID.toString).diagnostics.foreach(println)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def testNeverFails(): Unit = withSchema("xsd/never-fails-1.dfdl.xsd") { f =>
+ val good = UUID.randomUUID.toString
+ assertTrue(f.parse(good).validated)
+ }
+
+ @Test def testAlwaysFails(): Unit = withSchema("xsd/always-fails-1.dfdl.xsd") { f =>
+ val good = UUID.randomUUID.toString
+ assertFalse(f.parse(good).validated)
+ }
+
+ @Test def testExtends(): Unit = withSchema("xsd/extends-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("bob;l;smith").validated)
+ assertTrue(f.parse("bob;;smith").validated)
+ assertFalse(f.parse(";;smith").validated)
+ assertFalse(f.parse("bob;l;").validated)
+ assertFalse(f.parse(";l;").validated)
+ }
+
+ @Test def testNoNs1(): Unit = withSchema("xsd/without-ns-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("0;1", Always).validated)
+ assertFalse(f.parse("2;1").validated)
+ assertFalse(f.parse("0;0").validated)
+ }
+
+ @Test def testWithNs1(): Unit = withSchema("xsd/with-ns-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("0;1", Always).validated)
+ assertFalse(f.parse("2;1").validated)
+ assertFalse(f.parse("0;0").validated)
+ }
+}
Review comment:
I would also like to see testing of the CLI for this feature. First so that it ensures the CLI works and can find things the same way the API can, but also because it acts as a good example of expected usage for the CLI. Could you add some tests in daffodil-cli to test this functionality?
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/EmbeddedTesting.scala
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.validation.schematron
+
+import org.apache.daffodil.sapi.Daffodil
+import org.apache.daffodil.sapi.DataProcessor
+import org.apache.daffodil.sapi.Diagnostic
+import org.apache.daffodil.sapi.ParseResult
+import org.apache.daffodil.sapi.infoset.XMLTextInfosetOutputter
+import org.apache.daffodil.sapi.io.InputSourceDataInputStream
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.validation.schematron.SchSource.Xsd
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.File
+
+trait EmbeddedTesting {
+ def withJpgSchema(f: Validation => Unit): Unit = withSchema("xsd/jpeg.dfdl.xsd")(f)
+ def withBmpSchema(f: Validation => Unit): Unit = withSchema("xsd/bmp.dfdl.xsd")(f)
+ def withGifSchema(f: Validation => Unit): Unit = withSchema("xsd/gif.dfdl.xsd")(f)
+
+
+ case class PR(r: ParseResult) {
+ def validated: Boolean = !r.isValidationError()
+ def diagnostics: Seq[Diagnostic] = r.getDiagnostics
+ }
+
+ sealed trait PrintInfosetMode
+ case object Quiet extends PrintInfosetMode
+ case object AnyError extends PrintInfosetMode
+ case object ValError extends PrintInfosetMode
+ case object ProcError extends PrintInfosetMode
+ case object Always extends PrintInfosetMode
+
+ case class Validation(dp: DataProcessor) {
+ def parse(str: String, verbose: PrintInfosetMode = Quiet): PR = withBytes(str.getBytes, verbose)
+
+ def withBytes(bytes: Array[Byte], verbose: PrintInfosetMode = Quiet): PR = {
+ val bos = new ByteArrayOutputStream()
+ val r1 = dp.parse(
+ new InputSourceDataInputStream(new ByteArrayInputStream(bytes)),
+ new XMLTextInfosetOutputter(bos, true))
+
+// println(bos.toString)
Review comment:
Remove println
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560173582
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestValidating.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.regexp
+import org.junit.Test
+
+class TestValidating {
+ val data = "input/uuid.txt"
+ val uuid = "xsd/string.dfdl.xsd"
+ val never = "sch/never-fails.sch"
+ val always = "sch/always-fails.sch"
+
+ val alwaysResult = regexp("<.+-fails>2f6481e6-542c-11eb-ae93-0242ac130002</.+-fails>\n")
Review comment:
Ill try that. Looked like my paths were mangled in the logs, but maybe a red herring... I dont have a windows box to test so just going to take a few stabs at it and see what happens :boom: :hammer:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-757001356
Any remaining concerns with this? Work is complete unless there are changes requested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r559773725
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/TestEmbeddedSchematron.scala
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.validation.schematron
+
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbeddedSchematron extends EmbeddedTesting {
+ @Test def variation1(): Unit = withSchema("xsd/embedded-1.dfdl.xsd") { f =>
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def variation2(): Unit = withSchema("xsd/embedded-2.dfdl.xsd") { f =>
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def variation3(): Unit = withSchema("xsd/embedded-3.dfdl.xsd") { f =>
+ f.parse(UUID.randomUUID.toString).diagnostics.foreach(println)
+ assertTrue(f.parse(UUID.randomUUID.toString).validated)
+ f.parse(UUID.randomUUID.toString).diagnostics.foreach(println)
+ assertFalse(f.parse(UUID.randomUUID.toString.drop(1)).validated)
+ }
+
+ @Test def testNeverFails(): Unit = withSchema("xsd/never-fails-1.dfdl.xsd") { f =>
+ val good = UUID.randomUUID.toString
+ assertTrue(f.parse(good).validated)
+ }
+
+ @Test def testAlwaysFails(): Unit = withSchema("xsd/always-fails-1.dfdl.xsd") { f =>
+ val good = UUID.randomUUID.toString
+ assertFalse(f.parse(good).validated)
+ }
+
+ @Test def testExtends(): Unit = withSchema("xsd/extends-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("bob;l;smith").validated)
+ assertTrue(f.parse("bob;;smith").validated)
+ assertFalse(f.parse(";;smith").validated)
+ assertFalse(f.parse("bob;l;").validated)
+ assertFalse(f.parse(";l;").validated)
+ }
+
+ @Test def testNoNs1(): Unit = withSchema("xsd/without-ns-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("0;1", Always).validated)
+ assertFalse(f.parse("2;1").validated)
+ assertFalse(f.parse("0;0").validated)
+ }
+
+ @Test def testWithNs1(): Unit = withSchema("xsd/with-ns-1.dfdl.xsd") { f =>
+ assertTrue(f.parse("0;1", Always).validated)
+ assertFalse(f.parse("2;1").validated)
+ assertFalse(f.parse("0;0").validated)
+ }
+}
Review comment:
Done. This was a good exercise. :+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 merged pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 merged pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-750348383
If you are looking for negative tests, I recall the DFDL for JPEG without the schematron enforcement will literally accept /etc/termcap as a JPEG. Basically, any piece of line-noise bigger than some minimum is classified as a JPEG blob segment, and only the schematron rules enforce the proper positioning of these blob segments in a proper jpeg segment nest. I.e., the DFDL part does nothing to enforce the "grammar" rules of JPEG. So at least this one class of complete-line-noise negative examples for JPEG are easy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r542573243
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -30,13 +30,19 @@
<sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
</sch:rule>
</sch:pattern>
+ <sch:pattern name="test-short-uuid">
+ <sch:rule context="short-uuid">
+ <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
Review comment:
> I was expecting to see this rule as an annotation on the element declaration of short-uuid below.
The first pass had it embedded and it looked like it was going to get noisy.
I can see advantages of both.. will add an example back in for demonstration.
> W.r.t this test, I was hoping for a test using a sch rule that would use some relative paths like ../../foo/bar to reach around in the infoset and constrain disparate things together.
I see. Will add examples.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-743235168
### reference
- https://reference.niem.gov/niem/specification/sch-in-xsd/1.0alpha4/niem-schematron-in-xsd-1.0alpha4-2016-03-16.html
- https://journal.code4lib.org/articles/13546
- https://www.mulberrytech.com/papers/schematron-Philly.pdf
- http://schematronist.org/mailman/listinfo/schematron_schematronist.org
- https://xfront.com/awesome-power-of-schematron-plus-xpath2/index.html
- https://schematron.com/2018/12/standard-severity-levels-with-schematron-role/
- https://upg-dh.newtfire.org/explainSchematron.html
- http://dh.obdurodon.org/schematron-intro.xhtml
- https://github.com/lmlorenzo/mods-schema/blob/master/msul_mods_schema.xsd
- https://github.com/Schematron/schematron/blob/master/trunk/schematron/code/iso_schematron_skeleton_for_saxon.xsl#L60-L201
-
(some of these might make the readme, might not, here for posterity)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560170993
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/AnnotatedSchemaComponent.scala
##########
@@ -400,7 +400,12 @@ trait AnnotatedMixin { self: AnnotatedSchemaComponent =>
{
ai.attribute("source") match {
case None => {
- this.SDW(WarnID.AppinfoNoSource, """xs:appinfo without source attribute. Is source="http://www.ogf.org/dfdl/" missing?""")
+ // if a child node in the dfdl namespace exists we will provide a warning about using the source property
+ ai.child.flatMap(n => Option(n.namespace)).find{_.contains("dfdl")}.foreach { _ =>
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-748959533
@rthomas320
> DId the process create any SVRL files? Is a path to write the SVRL file to one of the commandline arguments?
I replied to this and some of the other comments on the dev list. Summary is no, but good idea and it should be able to be added.
> If SVRL reports were generated would it be possible to commit those to git also?
I can post the SVRL generated by a unit test or two here in the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560155294
##########
File path: daffodil-core/src/main/scala/org/apache/daffodil/dsom/AnnotatedSchemaComponent.scala
##########
@@ -400,7 +400,12 @@ trait AnnotatedMixin { self: AnnotatedSchemaComponent =>
{
ai.attribute("source") match {
case None => {
- this.SDW(WarnID.AppinfoNoSource, """xs:appinfo without source attribute. Is source="http://www.ogf.org/dfdl/" missing?""")
+ // if a child node in the dfdl namespace exists we will provide a warning about using the source property
+ ai.child.flatMap(n => Option(n.namespace)).find{_.contains("dfdl")}.foreach { _ =>
Review comment:
In the ``case Some`` below, we use contains("ogf") || contains("dfdl") to consider a DFDL namespace "acceptable". Should we use the same logic here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] create jira ticket
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] update LICENSE and NOTICE files appropriately
- [ ] tests with complex schematron
- [ ] move the gif, bmp, and jpeg schemas and tests
- [x] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] tests with complex schematron
- [ ] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
Capturing the initial poc here for review as it progresses.
The TestEmbeddedSchematron demonstrates the capability, but still a bit of work to do.
Based on #452
### draft todo
- [ ] create jira ticket
- [ ] detect sch or xsd
- [ ] doc in readme
- [ ] tests with complex schematron
- [ ] rebase after #452
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-748959533
@rthomas320
> The imbedded schematron appears to work well. DId the process create any SVRL files? Is a path to write the SVRL file to one of the commandline arguments?
I replied to this and some of the other comments on the dev list. Summary is no, but good idea and it should be able to be added.
> If SVRL reports were generated would it be possible to commit those to git also?
I can post the SVRL generated by a unit test or two here in the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549550340
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -24,25 +24,26 @@
<dfdl:format ref="GeneralFormat"/>
</xs:appinfo>
<xs:appinfo xmlns:sch="http://purl.oclc.org/dsdl/schematron">
- <sch:pattern name="test-uuid">
+ <sch:pattern name="validate-uuid">
<sch:rule context="uuid">
- <sch:assert test="string-length(.)=36">uuid must be 36 chars</sch:assert>
- <sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
- </sch:rule>
- </sch:pattern>
- <sch:pattern name="test-short-uuid">
- <sch:rule context="short-uuid">
- <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
+ <sch:assert test="string-length(time_low)=8"><sch:name/> wrong length</sch:assert>
Review comment:
> So you don't need package names on these functions? Why?
The default namespace is searched if the function is unprefixed, at least that is what I have gleaned from a couple random sources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r542516180
##########
File path: daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
##########
@@ -0,0 +1,107 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ based on an original transform by Eddie Robertsson
+ 2001/04/21 fn: added support for included schemas
+ 2001/06/27 er: changed XMl Schema prefix from xsd: to xs: and changed to the Rec namespace
+ 2010/04/14 rj: Update for ISO Schematron using xslt2
+-->
+
+<!--
+Open Source Initiative OSI - The MIT License:Licensing
+[OSI Approved License]
+
+This source code was previously available under the zlib/libpng license.
+Attribution is polite.
+
+The MIT License
+
+Copyright (c) 2004-2010 Rick Jellife and Academia Sinica Computing Centre, Taiwan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+-->
+<xsl:transform version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+xmlns:sch="http://purl.oclc.org/dsdl/schematron" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+ <!-- Set the output to be XML with an XML declaration and use indentation -->
+ <xsl:output method="xml" omit-xml-declaration="no" indent="yes" standalone="yes"/>
+ <!-- -->
+ <!-- match schema and call recursive template to extract included schemas -->
+ <!-- -->
+ <xsl:template match="xs:schema">
+ <!-- call the schema definition template ... -->
+ <xsl:call-template name="gatherSchema">
+ <!-- ... with current current root as the $schemas parameter ... -->
+ <xsl:with-param name="schemas" select="/"/>
+ <!-- ... and any includes in the $include parameter -->
+ <xsl:with-param name="includes"
+ select="document(/xs:schema/xs:*[self::xs:include or self::xs:import or self::xs:redefine]/@schemaLocation)"/>
+ </xsl:call-template>
+ </xsl:template>
+ <!-- -->
+ <!-- gather all included schemas into a single parameter variable -->
+ <!-- -->
+ <xsl:template name="gatherSchema">
+ <xsl:param name="schemas"/>
+ <xsl:param name="includes"/>
+ <xsl:choose>
+ <xsl:when test="count($schemas) < count($schemas | $includes)">
+ <!-- when $includes includes something new, recurse ... -->
+ <xsl:call-template name="gatherSchema">
+ <!-- ... with current $includes added to the $schemas parameter ... -->
+ <xsl:with-param name="schemas" select="$schemas | $includes"/>
+ <!-- ... and any *new* includes in the $include parameter -->
+ <xsl:with-param name="includes"
+ select="document($includes/xs:schema/xs:*[self::xs:include or self::xs:import or self::xs:redefine]/@schemaLocation)"/>
Review comment:
funny indentation stuff going on here.
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -30,13 +30,19 @@
<sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
</sch:rule>
</sch:pattern>
+ <sch:pattern name="test-short-uuid">
+ <sch:rule context="short-uuid">
+ <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
Review comment:
So this embeds the schematron rules in the schema, but they're still separate top-level things. That's useful, but not what I was thinking of.
I was expecting to see this rule as an annotation on the element declaration of short-uuid below. I think such a rule would not have a context attribute. The context would be implied by which specific element declaration.
W.r.t this test, I was hoping for a test using a sch rule that would use some relative paths like ../../foo/bar to reach around in the infoset and constrain disparate things together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560174117
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("2;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("0;0")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+}
Review comment:
Roger. Will clean this up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r548270010
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -24,25 +24,26 @@
<dfdl:format ref="GeneralFormat"/>
</xs:appinfo>
<xs:appinfo xmlns:sch="http://purl.oclc.org/dsdl/schematron">
- <sch:pattern name="test-uuid">
+ <sch:pattern name="validate-uuid">
<sch:rule context="uuid">
- <sch:assert test="string-length(.)=36">uuid must be 36 chars</sch:assert>
- <sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
- </sch:rule>
- </sch:pattern>
- <sch:pattern name="test-short-uuid">
- <sch:rule context="short-uuid">
- <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
+ <sch:assert test="string-length(time_low)=8"><sch:name/> wrong length</sch:assert>
Review comment:
Ive saw it done both ways. Will try to pin down why with a reference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-750345274
Tests fixed, going to work through my checklist above. BMP, GIF, and JPEG appear to be working... I need a generator for each of them to created mutated versions of each... :christmas_tree: :gift:
For now though things are close and I can reshape/add examples based on the feedback above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-751933010
@rthomas320
> If SVRL reports were generated would it be possible to commit those to git also?
A few examples of svrl output
```
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_low"/>
<svrl:failed-assert location="/uuid[1]/time_low[1]" test="string-length(.)=8">
<svrl:text>time_low wrong length</svrl:text>
</svrl:failed-assert>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_mid"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_hi_and_version"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="clock_seq_hi_and_res"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="node"/>
</svrl:schematron-output>
```
```
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="first"/>
<svrl:failed-assert location="/name[1]/first[1]" test="string-length(.) > 0">
<svrl:text>first is blank.</svrl:text>
</svrl:failed-assert>
<svrl:fired-rule context="last"/>
<svrl:failed-assert location="/name[1]/last[1]" test="string-length(.) > 0">
<svrl:text>last is blank.</svrl:text>
</svrl:failed-assert>
</svrl:schematron-output>
```
```
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="first"/>
<svrl:fired-rule context="last"/>
</svrl:schematron-output>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r541006190
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ elementFormDefault="qualified">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat" />
+ </xs:appinfo>
+ </xs:annotation>
+
+ <xs:element name="uuid" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator="%NL;">
Review comment:
Updated. Is that change enough to cover that the rule is properly scoped to the context?
Working on improving my daffodil schema understanding at the same time as adding more complex tests and working with the JPEG schema... :turtle:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 merged pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 merged pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560227156
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/package.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil
+
+import org.apache.daffodil.CLI.Util
+import net.sf.expectit.matcher.Matchers.eof
+import net.sf.expectit.Result
+import net.sf.expectit.matcher.Matcher
+import net.sf.expectit.matcher.Matchers.contains
+import net.sf.expectit.matcher.Matchers.regexp
+import org.junit.Assert.fail
+
+import java.io.File
+import java.io.FileOutputStream
+import java.nio.file.Path
+import java.nio.file.Paths
+import scala.util.Failure
+import scala.util.Success
+import scala.util.Try
+
+package object schematron {
+ val FailureErrorCodeResult = 1
+
+ /**
+ * executes a command in a shell with the provided expectations and error code using a mustache syntax looks up files
+ * from local resources {path} or the daffodil-schematron resources {{path}}
+ * @param ec expected error code
+ * @param stderr join stderr in output
+ * @param body 2 tuple of daffodil arguments and expectation
+ */
+ def withShell[R <: Result](ec: Int = 0, stderr: Boolean = false)(body: => (String, Matcher[R])): Unit = {
+ val (argstring, expectation) = body
+ val args = mustache.replaceAllIn(argstring, _ match {
+ case mustache2(p) => schPath(p).toString
+ case mustache1(p) => cliPath(p).toString
+ })
+
+ val joinStdErr = if(stderr) "2>&1" else ""
+ val cmd = Util.binPath :: args :: joinStdErr :: Nil mkString " "
+ val shell = Util.start("")
+ try {
+ shell.sendLine(cmd).expect(expectation)
+
+ val actualEc = shell.sendLine(echoEC).expect(contains("\n")).getBefore
+ Try(actualEc.toInt) match {
+ case Success(`ec`) => // good
+ case Success(v) => fail(s"wrong ec, $v")
+ case Failure(_) => fail(s"unparseable ec, $actualEc")
+ }
+
+ shell.send("exit\n")
+ shell.expect(eof)
+ shell.close()
+ } finally {
+ shell.close()
+ }
+ }
+
+ // two reasons for this bit of parsing indirection
+ // 1. support consuming resources from multiple projects
+ // 2. avoid doing the noisy and repetetive resolution in the unit tests
+ private val mustache = """\{{1,2}(.+?)}{1,2}""".r.unanchored
+ private val mustache1 = """\{(.+?)}""".r.unanchored
+ private val mustache2 = """\{\{(.+?)}}""".r.unanchored
+
+ private def schPath(p: String): Path = path(s"daffodil-schematron/src/test/resources/$p")
+ private def cliPath(p: String): Path = path(s"daffodil-cli/src/it/resources/org/apache/daffodil/CLI/$p")
+ private def path(p: String): Path = {
+ val daffpath = Util.daffodilPath(p)
+ Paths.get(if(Util.isWindows) Util.cmdConvert(daffpath) else daffpath).toAbsolutePath
+ }
+ private lazy val echoEC: String = s"echo ${if(Util.isWindows) "%errorlevel%" else "$?"}"
+
+ /**
+ * number of lines, any content on those lines
+ * @param n line count
+ * @return
+ */
+ def anyLines(n: Int): Matcher[_] = regexp(Seq.fill(n)(".+\n").mkString)
+
+ /**
+ * make a temp file containing the bytes
+ * @param str
+ * @return
+ */
+ def mktmp(d: Array[Byte]): Path = {
+ val f = File.createTempFile("schval", "data")
+ val os = new FileOutputStream(f)
+ os.write(d)
+ os.close()
+ f.toPath
+ }
+ def mktmp(str: String): Path = mktmp(str.getBytes)
+}
Review comment:
Yeah, needs more work, consider it a reference implementation. I added a comment for DAFFODIL-2381 on the package object, will comment on the jira ticket.
> (and honestly, not very obvious)
Noted this behavior in the doc on the package object.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] rthomas320 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
rthomas320 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r547992297
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
XPath allows the greater than '>' symbol, but not the less than '<' symbol
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-764833052
Windows issues fixed up, squashed and rebased. Ready to go I believe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560162148
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestValidating.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.regexp
+import org.junit.Test
+
+class TestValidating {
+ val data = "input/uuid.txt"
+ val uuid = "xsd/string.dfdl.xsd"
+ val never = "sch/never-fails.sch"
+ val always = "sch/always-fails.sch"
+
+ val alwaysResult = regexp("<.+-fails>2f6481e6-542c-11eb-ae93-0242ac130002</.+-fails>\n")
Review comment:
Tests in windows are failing, perhaps due to this \n? Is this \r\n on windows? Can we just make this tests loser so it doesn't care about the newline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560225908
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("2;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("0;0")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+}
Review comment:
Yes. What strted the entire abstraction of temp file creation was to set deleteOnExit() for all these, and I ended up forgetting it :laughing:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r555131072
##########
File path: daffodil-schematron/src/main/resources/META-INF/LICENSE
##########
@@ -237,3 +237,28 @@ subcomponents is subject to the terms and conditions of the following licenses.
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+
+ This product bundles content from the Schematron converters, including
+ the following files:
+ - daffodil-schematron/src/main/resources/iso-schematron-xslt2/ExtractSchFromXSD-2.xsl
Review comment:
:ok_hand:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r530616792
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ elementFormDefault="qualified">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat" />
+ </xs:appinfo>
+ </xs:annotation>
+
+ <xs:element name="uuid" type="xs:string" dfdl:lengthKind="delimited" dfdl:terminator="%NL;">
Review comment:
This test is not able to distinguish "." from "/", which is a key thing for tests to cover. Nest this uuid element inside another root element and then this would have more coverage, or add a second such test which would test exactly the issue of whether "." reflects the right infoset location in the schematron rules.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r549421504
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
XPath 1.0 seems to not require a correct namespace on function names. XPath 2.0 does.
per: https://www.w3.org/2006/xpath-functions/
DFDL's expression language is similar to XPath 2.0. E.g., you cannot just type string-length(../some/path) in DFDL nor XPath 2.0 and expect it to work. You must bind a prefix (usually "fn" to namespace http://www.w3.org/2005/xpath-functions/ and use the prefix when calling the functions.
If the current schematron is limited to XPath 1.0, then so be it for now. But there's a chance it is using XPath 2.0 but turning on the XPath 1.0 compatibility mode. In which case, if we could *not* do that, then the expressions would be more harmonious with the DFDL expression language.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560181520
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/package.scala
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil
+
+import org.apache.daffodil.CLI.Util
+import net.sf.expectit.matcher.Matchers.eof
+import net.sf.expectit.Result
+import net.sf.expectit.matcher.Matcher
+import net.sf.expectit.matcher.Matchers.contains
+import net.sf.expectit.matcher.Matchers.regexp
+import org.junit.Assert.fail
+
+import java.io.File
+import java.io.FileOutputStream
+import java.nio.file.Path
+import java.nio.file.Paths
+import scala.util.Failure
+import scala.util.Success
+import scala.util.Try
+
+package object schematron {
+ val FailureErrorCodeResult = 1
+
+ /**
+ * executes a command in a shell with the provided expectations and error code using a mustache syntax looks up files
+ * from local resources {path} or the daffodil-schematron resources {{path}}
+ * @param ec expected error code
+ * @param stderr join stderr in output
+ * @param body 2 tuple of daffodil arguments and expectation
+ */
+ def withShell[R <: Result](ec: Int = 0, stderr: Boolean = false)(body: => (String, Matcher[R])): Unit = {
+ val (argstring, expectation) = body
+ val args = mustache.replaceAllIn(argstring, _ match {
+ case mustache2(p) => schPath(p).toString
+ case mustache1(p) => cliPath(p).toString
+ })
+
+ val joinStdErr = if(stderr) "2>&1" else ""
+ val cmd = Util.binPath :: args :: joinStdErr :: Nil mkString " "
+ val shell = Util.start("")
+ try {
+ shell.sendLine(cmd).expect(expectation)
+
+ val actualEc = shell.sendLine(echoEC).expect(contains("\n")).getBefore
+ Try(actualEc.toInt) match {
+ case Success(`ec`) => // good
+ case Success(v) => fail(s"wrong ec, $v")
+ case Failure(_) => fail(s"unparseable ec, $actualEc")
+ }
+
+ shell.send("exit\n")
+ shell.expect(eof)
+ shell.close()
+ } finally {
+ shell.close()
+ }
+ }
+
+ // two reasons for this bit of parsing indirection
+ // 1. support consuming resources from multiple projects
+ // 2. avoid doing the noisy and repetetive resolution in the unit tests
+ private val mustache = """\{{1,2}(.+?)}{1,2}""".r.unanchored
+ private val mustache1 = """\{(.+?)}""".r.unanchored
+ private val mustache2 = """\{\{(.+?)}}""".r.unanchored
+
+ private def schPath(p: String): Path = path(s"daffodil-schematron/src/test/resources/$p")
+ private def cliPath(p: String): Path = path(s"daffodil-cli/src/it/resources/org/apache/daffodil/CLI/$p")
+ private def path(p: String): Path = {
+ val daffpath = Util.daffodilPath(p)
+ Paths.get(if(Util.isWindows) Util.cmdConvert(daffpath) else daffpath).toAbsolutePath
+ }
+ private lazy val echoEC: String = s"echo ${if(Util.isWindows) "%errorlevel%" else "$?"}"
+
+ /**
+ * number of lines, any content on those lines
+ * @param n line count
+ * @return
+ */
+ def anyLines(n: Int): Matcher[_] = regexp(Seq.fill(n)(".+\n").mkString)
+
+ /**
+ * make a temp file containing the bytes
+ * @param str
+ * @return
+ */
+ def mktmp(d: Array[Byte]): Path = {
+ val f = File.createTempFile("schval", "data")
+ val os = new FileOutputStream(f)
+ os.write(d)
+ os.close()
+ f.toPath
+ }
+ def mktmp(str: String): Path = mktmp(str.getBytes)
+}
Review comment:
This is great! But it fells like it belongs in ``CLI/Util.scala`` for other tests to use. Though, I guess there are lots of little bits that at are very specific to schematron tests, so maybe it's not quite ready? For example, single curly braces means the resource is in daffodil-schematron is very specific to this (and honestly, not very obvious).
Something like this is definitely needed, and is almost certainly how DAFFODIL-2381 should be resolved.
However, I don't really want there to be two different ways to write CLI tests long term, so we need to make sure we document that this is written differently from others CLI tests, and that that should probably be fixed as part of DAFFODIL-2381. Perhaps add a comment to DAFFODIL-2381 to specifically point out this code, and explain that this should become the basis for those changes, and maybe mention anything that is specific to schematron that likely needs to be replaced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-743235168
### reference
- https://reference.niem.gov/niem/specification/sch-in-xsd/1.0alpha4/niem-schematron-in-xsd-1.0alpha4-2016-03-16.html
- https://journal.code4lib.org/articles/13546
- https://www.mulberrytech.com/papers/schematron-Philly.pdf
- http://schematronist.org/mailman/listinfo/schematron_schematronist.org
- https://xfront.com/awesome-power-of-schematron-plus-xpath2/index.html
- https://schematron.com/2018/12/standard-severity-levels-with-schematron-role/
- https://upg-dh.newtfire.org/explainSchematron.html
- http://dh.obdurodon.org/schematron-intro.xhtml
- https://github.com/lmlorenzo/mods-schema/blob/master/msul_mods_schema.xsd
-
(some of these might make the readme, might not, here for posterity)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-757001356
Any remaining concerns with this? Work is complete unless there are changes requested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-751933010
@rthomas320
> If SVRL reports were generated would it be possible to commit those to git also?
A few examples of svrl output
```xml
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_low"/>
<svrl:failed-assert location="/uuid[1]/time_low[1]" test="string-length(.)=8">
<svrl:text>time_low wrong length</svrl:text>
</svrl:failed-assert>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_mid"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="time_hi_and_version"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="clock_seq_hi_and_res"/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="node"/>
</svrl:schematron-output>
```
```xml
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="first"/>
<svrl:failed-assert location="/name[1]/first[1]" test="string-length(.) > 0">
<svrl:text>first is blank.</svrl:text>
</svrl:failed-assert>
<svrl:fired-rule context="last"/>
<svrl:failed-assert location="/name[1]/last[1]" test="string-length(.) > 0">
<svrl:text>last is blank.</svrl:text>
</svrl:failed-assert>
</svrl:schematron-output>
```
```xml
<svrl:schematron-output title="" schemaVersion="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:schold="http://www.ascc.net/xml/schematron" xmlns:saxon="http://saxon.sf.net/" xmlns:iso="http://purl.oclc.org/dsdl/schematron" xmlns:svrl="http://purl.oclc.org/dsdl/svrl">
<svrl:active-pattern document=""/>
<svrl:active-pattern document=""/>
<svrl:fired-rule context="first"/>
<svrl:fired-rule context="last"/>
</svrl:schematron-output>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] mbeckerle commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
mbeckerle commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r547488352
##########
File path: daffodil-schematron/src/test/resources/xsd/embedded-1.dfdl.xsd
##########
@@ -24,25 +24,26 @@
<dfdl:format ref="GeneralFormat"/>
</xs:appinfo>
<xs:appinfo xmlns:sch="http://purl.oclc.org/dsdl/schematron">
- <sch:pattern name="test-uuid">
+ <sch:pattern name="validate-uuid">
<sch:rule context="uuid">
- <sch:assert test="string-length(.)=36">uuid must be 36 chars</sch:assert>
- <sch:assert test="substring(.,9,1)='-' and substring(.,14,1)='-' and substring(.,19,1)='-' and substring(.,24,1)='-'">malformed uuid</sch:assert>
- </sch:rule>
- </sch:pattern>
- <sch:pattern name="test-short-uuid">
- <sch:rule context="short-uuid">
- <sch:assert test="string-length(.)=8">short uuid must be 8 chars</sch:assert>
+ <sch:assert test="string-length(time_low)=8"><sch:name/> wrong length</sch:assert>
Review comment:
So you don't need package names on these functions? Why?
I would expect to see "fn:string-length(.) eq 8"
Is schematron's expression language actually not XPath based?
##########
File path: daffodil-schematron/src/test/scala/org/apache/daffodil/validation/schematron/TestBasicValidation.scala
##########
@@ -68,7 +68,7 @@ class TestBasicValidation {
val factory = new TransformerFactoryImpl()
factory.setURIResolver(resolver)
val sch = getClass.getClassLoader.getResourceAsStream("sch/schematron-1.sch")
- val rules = Transforms.fromSch(sch, factory)
+ val rules = Transforms.from(sch, Sch, factory)
Review comment:
Might be a separate ticket, but it seems to me we want to be able to write TDML tests with validation settings that will cause them to invoke schematron as part of the runtime. E.g., define a tdml:testSuite with defaultValidation="schematron" or a specific tdml:parserTestCase with validation="schematron"
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
I expected the test expression to have prefixes like fn:string-length(.)
Is the schematron expression language not XPath based? I am also surprised to see ">" as an operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560172827
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
Review comment:
Some of these tests check for FailureErrorCodeResults, but looked like they are checking for an infoset? Does that mean these tests fail validation? Is it worth also checking for a validation error in that case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] stevedlawrence commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
stevedlawrence commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r560182696
##########
File path: daffodil-cli/src/it/scala/org/apache/daffodil/schematron/TestEmbedded.scala
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.schematron
+
+import net.sf.expectit.matcher.Matchers.sequence
+import net.sf.expectit.matcher.Matchers.contains
+import org.junit.Test
+
+import java.util.UUID
+
+class TestEmbedded {
+ @Test def alwaysFails(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(UUID.randomUUID.toString)
+ val schema = "xsd/always-fails-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> anyLines(2)
+ }
+
+ @Test def unitPriceWithoutValidation(): Unit = withShell() {
+ val data = mktmp("widget,monday,1,$5.00,$5.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list -s {{$schema}} $data" -> contains("</ex:list>\n")
+ }
+
+ @Test def unitPriceWithValidation(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,1,$5.00,$6.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ anyLines(3)
+ )
+ }
+
+ @Test def unitPriceWithValidationCheckMessage(): Unit = withShell(FailureErrorCodeResult, true) {
+ val data = mktmp("widget,monday,5,$5.00,$25.00||gadget,tuesday,1,$10.00,$11.00")
+ val schema = "xsd/unit_price.dfdl.xsd"
+ s"parse -r list --validate schematron={{$schema}} -s {{$schema}} $data" -> sequence(
+ contains("</ex:list>\n"),
+ contains("[error] Validation Error: wrong unit price for gadget, tuesday\n"),
+ anyLines(2)
+ )
+ }
+
+ @Test def extends1(): Unit = withShell() {
+ val data = mktmp("bob;l;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends2(): Unit = withShell() {
+ val data = mktmp("ob;;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";;smith")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends4(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("bob;l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def extends5(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp(";l;")
+ val schema = "xsd/extends-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</name>\n")
+ }
+
+ @Test def testWithNs1(): Unit = withShell() {
+ val data = mktmp("0;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs2(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("2;1")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+
+ @Test def testWithNs3(): Unit = withShell(FailureErrorCodeResult) {
+ val data = mktmp("0;0")
+ val schema = "xsd/with-ns-1.dfdl.xsd"
+ s"parse --validate schematron={{$schema}} -s {{$schema}} $data" -> contains("</myns:interval>\n")
+ }
+}
Review comment:
Can you delete these tmp files when a test is finished? Polluting tmp and not cleaning it up is something Daffodil tests are really bad at, and we have dozens of different places and ways we create temp files. Some we need to be better at.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on a change in pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on a change in pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#discussion_r548263498
##########
File path: daffodil-schematron/src/test/resources/xsd/extends-1.dfdl.xsd
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:sch="http://purl.oclc.org/dsdl/schematron">
+
+ <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+ <xs:annotation>
+ <xs:appinfo source="http://www.ogf.org/dfdl/">
+ <dfdl:format ref="GeneralFormat"/>
+ </xs:appinfo>
+ <xs:appinfo>
+ <sch:pattern>
+ <sch:rule abstract="true" id="blank">
+ <sch:assert test="string-length(.) > 0"><sch:name/> is blank.</sch:assert>
Review comment:
All the references I see say there is a full set of operators in both xpath 1 and 2. Here is one of several without looking too hard https://www.w3schools.com/xml/xpath_operators.asp
What I am missing in this discussion about limited xpath operators (besides much experience with xpath)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 commented on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 commented on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-764833052
Windows issues fixed up, squashed and rebased. Ready to go I believe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-daffodil] jw3 edited a comment on pull request #463: Embedded Schematron
Posted by GitBox <gi...@apache.org>.
jw3 edited a comment on pull request #463:
URL: https://github.com/apache/incubator-daffodil/pull/463#issuecomment-733411017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org