You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Taylor Wise <tw...@tresys.com> on 2017/11/17 18:13:12 UTC
Unordered Sequence and Choice Branch Violations
While executing some existing tests I've come across a case that I need guidance on.
Given the following test case:
<tdml:defineSchema name="simple_invalid_path_to_branch">
<dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
lengthKind="delimited" />
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="unordered"
dfdl:separator=",">
<xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="a">
</xs:element>
<xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="b">
</xs:element>
<xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="c">
</xs:element>
<xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
dfdl:inputValueCalc="{ ../ex:e3 }">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<tdml:parserTestCase name="test_simple_invalid_path_to_branch"
model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
root="USG_01" roundTrip="false">
<tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
<tdml:errors>
<tdml:error>Schema Definition Error</tdml:error>
<tdml:error>Expression</tdml:error>
<tdml:error>../e3</tdml:error>
<tdml:error>navigates to another branch</tdml:error>
<tdml:error>e3</tdml:error>
</tdml:errors>
</tdml:parserTestCase>
The result of the test is the following:
edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
Something like:
<xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
<xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
dfdl:occursCountKind="parsed">
<xs:complexType>
<xs:choice dfdl:choiceLengthKind="implicit">
<xs:element name="e1" type="xs:string" dfdl:initiator="a" />
<xs:element name="e2" type="xs:int" dfdl:initiator="b" />
<xs:element name="e3" type="xs:string" dfdl:initiator="c" />
<xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
</xs:choice>
</xs:complexType>
</xs:element>
</xs:sequence>
So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
Offending code:
final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
val branchesOk = groupMembersNoRefs map { branch =>
if (!branch.isRepresented) {
schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
false
} else true
}
assuming(branchesOk.forall { x => x })
}.value
About IVC from the spec:
It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
It is not possible to place this property in scope on a dfdl:format annotation.
This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
Re: Unordered Sequence and Choice Branch Violations
Posted by Mike Beckerle <mb...@tresys.com>.
I think this is pretty clear - your combinator must implement the complete functionality, not be a combnator that combines a repeat combinator and a choice combinator.
Your combinator should take the gram objects for what would have been the branches of the choice, and combine them in the repeating pattern and choice pattern needed.
The combinator will be a bit different, but a lot like taking the current unbounded rep parser, and inlining the alt comp parser code into it.
________________________________
From: Taylor Wise
Sent: Wednesday, December 6, 2017 9:46:48 AM
To: Steve Lawrence; dev@daffodil.apache.org; Mike Beckerle
Subject: Re: Unordered Sequence and Choice Branch Violations
So having implemented this approach, Steve, I've run into a bit of a snafu.
What I've done is that I modified the UnorderedSequenceCombinator to use the AltComp and RepUnbounded parsers rather than the hacked method we used before.
This looks like the following:
case class UnorderedSequenceCombinator(s: Sequence, rawTerms: Seq[Gram])
extends Terminal(s, !rawTerms.filterNot { _.isEmpty }.isEmpty) {
private val mt: Gram = EmptyGram
lazy val body = rawTerms.foldRight(mt) { _ ~ _ }
lazy val terms = rawTerms.filterNot { _.isEmpty }
lazy val unparsers = terms.map { _.unparser }.toVector
lazy val parsers = terms.map { _.parser }
...
lazy val erd = s.runtimeData.immediateEnclosingElementRuntimeData.get
lazy val choice = new AltCompParser(context.runtimeData, parsers)
lazy val rep = new RepUnboundedParser(OccursCountKind.Parsed, choice, erd) // Unordered is a RepUnbounded of Choice.
Console.out.println(rep)
lazy val parser: DaffodilParser = new UnorderedSequenceCombinatorParser(s.termRuntimeData, rep, sortOrder, scalarMembers)
}
class UnorderedSequenceCombinatorParser(rd: RuntimeData, bodyParser: Parser,
sortOrder: Seq[(NamedQName, NS)],
scalarMembers: Seq[(NamedQName, NS)])
extends ParserObject(rd) {
override def nom = "UnorderedSequence"
override lazy val childProcessors = Seq(bodyParser)
...
def parse(start: PState): Unit = {
start.mpstate.groupIndexStack.push(1L) // one-based indexing
bodyParser.parse1(start) // Body parser here should be the RepUnbounded of Choice
val elt = start.thisElement
val complex = elt.asComplex
// Removes 'choiceElement'
//val childrenDetached = complex.childNodes.flatMap(n => n.contents.flatMap(c => c.contents))
val childrenDetached = complex.childNodes
checkScalarsOccurExactlyOnce(childrenDetached, start)
// dropDefaulted(currentElemAfter, end)
checkOccurrence(elt, childrenDetached, start)
// Sort so that the contents are in the expected order.
sort(complex, childrenDetached, start)
start.mpstate.groupIndexStack.pop()
start.mpstate.moveOverOneGroupIndexOnly()
()
}
}
This allows for the same functionality as the hack of mutating the XML of an unordered sequence into a sequence of choice.
If you look at the following test, just simply using a combination of the appropriate parsers is not enough.
<tdml:defineSchema name="simple_min_max_occurs">
<dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
lengthKind="implicit" />
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="unordered">
<xs:element name="e1" type="ex:stNumStr"
dfdl:lengthKind="explicit" dfdl:length="2" minOccurs="1"
maxOccurs="2" dfdl:occursCountKind="parsed">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="e2" type="ex:stHexStr"
dfdl:lengthKind="explicit" dfdl:length="2">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="e3" type="xs:string" dfdl:lengthKind="explicit"
dfdl:length="2">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:simpleType name="stHexStr">
<xs:restriction base="xs:string">
<xs:pattern value="[a-fA-F0-9]+"></xs:pattern>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="stNumStr">
<xs:restriction base="xs:string">
<xs:pattern value="[0-9]+"></xs:pattern>
</xs:restriction>
</xs:simpleType>
</tdml:defineSchema>
<tdml:parserTestCase name="test_simple_min_max_occurs"
model="simple_min_max_occurs" description="Section 14 Sequence group with left over data - DFDL-14-001R"
root="USG_01" validation="on">
<tdml:document><![CDATA[veba0022]]></tdml:document>
<tdml:infoset>
<tdml:dfdlInfoset>
<USG_01>
<e1>00</e1>
<e1>22</e1>
<e2>ba</e2>
<e3>ve</e3>
</USG_01>
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
Why? Because 'parsers' in the code above looks like the following:
List(<Array><RepUnbounded name='e1' n='-1'><Element name='e1'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e1 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element></RepUnbounded></Array>, <Element name='e2'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e2 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element>, <Element name='e3'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e3 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element>)
The problem here is that it wants to do a RepUnbounded for e1. This firmly expects to encounter e1 1 to 2 times. Because we actually encounter 've' first this RepUnbounded for e1 fails but exits with a failure suppressed. This then basically tells the AltComp that the choice was successful and causes the parse to end prematurely with the following error:
edu.illinois.ncsa.daffodil.tdml.TDMLException: Runtime Schema Definition Error: {http://example.com}e2 is scalar but occurred 0 times.
Path: http://example.com
Offending children:
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
Parse Error: RepUnbounded - No forward progress at byte 0. Attempt to parse ex:USG_01 succeeded but consumed no data.
Please re-examine your schema to correct this infinite loop.
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
Validation Error: e1 occurred '0' times when it was expected to be a minimum of '1' and a maximum of '2' times.
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
I believe the fix for this would be to alter the grammar in the case of unordered sequences. Meaning that at the point where we decide whether the termContentBody of the LocalElement is
an enclosedElement or recurrance, we need to make it an enclosedElement if it's the immediate child of an unordered sequence.
Thoughts?
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Thursday, November 30, 2017 4:18:35 PM
To: Taylor Wise; dev@daffodil.apache.org; Mike Beckerle
Subject: Re: Unordered Sequence and Choice Branch Violations
I think we should just remove the choiceElement entirely. A new
UnorderedSequenceCombinatorParser should be able to handle all the logic
that is currently handled by converting unordered sequences to a
sequence of choices (which looks to be done in the lazy val unorderedSeq
in SequenceGroup.scala). It does seem like this expressions accessing
other branches of an unordered sequences should be an SDE.
On 11/30/2017 03:39 PM, Taylor Wise wrote:
> By this same token then, any assertions/expressions that navigate to a different
> branch of the unordered sequence/choice are invalid.
>
>
> "When processing a choice group the parser validates any contained path
> expressions. If a path expression contained inside a choice branch refers to any
> other branch of the choice, then it is a schema definition error. Note that this
> rule handles nested choices also. A path that navigates outward from an inner
> choice to another alternative of an outer choice is violating this rule with
> respect to the outer choice."
>
>
> I'm currently running into an issue where there's an existing test that has an
> expression in the initiator of an element in an unordered sequence. The
> expression refers to a prior element. Because of the spec, this should SDE.
>
>
> However, it's currently failing stating that the element does not exist when it
> should have already been added. This is leading me to believe that there is
> either something funky going on with the expression steps due to the additional
> choiceElement or that somehow the element (e3) isn't being kept around in the
> infoset (unlikely as tests without expressions run fine).
>
>
> The existing test:
>
>
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="unordered"
> dfdl:separator=",">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
> model="simple_invalid_path_to_branch" description="Section 14 Sequence group
> with left over data - DFDL-14-001R"
> root="USG_01" roundTrip="false">
> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
> <tdml:errors>
> <tdml:error>Schema Definition Error</tdml:error>
> <tdml:error>Expression</tdml:error>
> <tdml:error>../e3</tdml:error>
> <tdml:error>navigates to another branch</tdml:error>
> <tdml:error>e3</tdml:error>
> </tdml:errors>
> </tdml:parserTestCase>
> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message
> "../e3" in any of the actual diagnostic messages:
> Runtime Schema Definition Error: Expression Evaluation Error: Child element
> {http://example.com}e3 does not exist.
> Schema context: e4 Location line 199 column 22 in
> file:/tmp/simple_invalid_path_to_branch1343627752058513822.dfdl.xsd
> Data location was preceding byte 12 limit(bytes) 19
>
> The source-to-source transformation:
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="ordered"
> dfdl:separator=",">
> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
> occursCountKind="parsed">
> <xs:complexType>
> <xs:choice dfdl:choiceLengthKind="implicit">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:choice>
> </xs:complexType>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> In this case, e3 should definitely exist by the time the choice alternative for
> e4 is run. Stepping through, I can see each alternative succeed for each element
> until it tries e4.
> At which time, it doesn't appear that e3 is in the infoset (though it should
> be). What appears to be happening is that each time an alternative succeeds,
> a choiceElement (containing the parsed element) is added to USG_01. So I think
> what's happening is that these choiceElement's are interfering with the
> expression evaluations.
>
> You end up with something like this in the Infoset:
> <USG_01>
> <choiceElement><e2>2</e2></choiceElement>
> <choiceElement><e1>1</e1></choiceElement>
> <choiceElement><e1>4</e1></choiceElement>
> <choiceElement><e3>3</e3></choiceElement>
> <choiceElement> ....Let's look for the Initiator from e3... </choiceElement>
> </USG_01>
>
> The expression is expecting to be able to go UP to USG_01 and ask for a child
> named e3. But, it would appear that it's going UP to its choiceElement and
> asking for e3.
> Which does not exist.
>
> Which brings me to two possible solutions:
>
> 1. Somehow detect that we're running an expression from within a choice and
> 'hide/skip' choiceElements. Likely meaning we'll have to prepend an extra
> ../ somewhere. This adds some unnecessary complexity to the expression
> code. Which I kind of want to avoid.
> 2. Remove the choiceElement entirely.
>
>
>
>
>
>
> --------------------------------------------------------------------------------
> *From:* Steve Lawrence <sl...@apache.org>
> *Sent:* Friday, November 17, 2017 1:37:46 PM
> *To:* dev@daffodil.apache.org; Taylor Wise; Mike Beckerle
> *Subject:* Re: Unordered Sequence and Choice Branch Violations
> On 11/17/2017 01:13 PM, Taylor Wise wrote:
>> While executing some existing tests I've come across a case that I need guidance on.
>>
>>
>> Given the following test case:
>>
>> <tdml:defineSchema name="simple_invalid_path_to_branch">
>> <dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
>> lengthKind="delimited" />
>> <xs:element name="USG_01">
>> <xs:complexType>
>> <xs:sequence dfdl:sequenceKind="unordered"
>> dfdl:separator=",">
>>
>> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="a">
>> </xs:element>
>>
>> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="b">
>> </xs:element>
>>
>> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="c">
>> </xs:element>
>>
>> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
>> dfdl:inputValueCalc="{ ../ex:e3 }">
>> </xs:element>
>>
>> </xs:sequence>
>> </xs:complexType>
>> </xs:element>
>>
>>
>> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
>> model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
>> root="USG_01" roundTrip="false">
>> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
>> <tdml:errors>
>> <tdml:error>Schema Definition Error</tdml:error>
>> <tdml:error>Expression</tdml:error>
>> <tdml:error>../e3</tdml:error>
>> <tdml:error>navigates to another branch</tdml:error>
>> <tdml:error>e3</tdml:error>
>> </tdml:errors>
>> </tdml:parserTestCase>
>>
>> The result of the test is the following:
>>
>> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
>> Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
>> Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
>>
>> Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
>>
>> Something like:
>>
>> <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
>>
>> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
>> dfdl:occursCountKind="parsed">
>>
>> <xs:complexType>
>>
>> <xs:choice dfdl:choiceLengthKind="implicit">
>>
>> <xs:element name="e1" type="xs:string" dfdl:initiator="a" />
>>
>> <xs:element name="e2" type="xs:int" dfdl:initiator="b" />
>>
>> <xs:element name="e3" type="xs:string" dfdl:initiator="c" />
>>
>> <xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
>>
>> </xs:choice>
>>
>> </xs:complexType>
>>
>> </xs:element>
>>
>> </xs:sequence>
>>
>> So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
>> The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
>>
>> Offending code:
>> final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
>> val branchesOk = groupMembersNoRefs map { branch =>
>> if (!branch.isRepresented) {
>> schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
>> false
>> } else true
>> }
>> assuming(branchesOk.forall { x => x })
>> }.value
>>
>>
>> About IVC from the spec:
>> It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
>> It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
>> It is not possible to place this property in scope on a dfdl:format annotation.
>> This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
>> If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
>> If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
>> If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
>>
>
> Seems to me like this shouldn't be allowed to me. I don't think
> inputValueCalc elements even consume any data, so it doesn't make sense
> to be in an unordered sequence. Otherwise you could parse
> inputValueCalc's forever as part of that sequence.
>
> I'll also add that if we add a new UnordredSequenceCombinator, I would
> think we could get rid of the code that creates a fake sequence of
> choices. That always seemed a bit of a hack and made things awkward, for
> example the "choiceElement" showing up in the path.
>
Re: Unordered Sequence and Choice Branch Violations
Posted by Taylor Wise <tw...@tresys.com>.
So having implemented this approach, Steve, I've run into a bit of a snafu.
What I've done is that I modified the UnorderedSequenceCombinator to use the AltComp and RepUnbounded parsers rather than the hacked method we used before.
This looks like the following:
case class UnorderedSequenceCombinator(s: Sequence, rawTerms: Seq[Gram])
extends Terminal(s, !rawTerms.filterNot { _.isEmpty }.isEmpty) {
private val mt: Gram = EmptyGram
lazy val body = rawTerms.foldRight(mt) { _ ~ _ }
lazy val terms = rawTerms.filterNot { _.isEmpty }
lazy val unparsers = terms.map { _.unparser }.toVector
lazy val parsers = terms.map { _.parser }
...
lazy val erd = s.runtimeData.immediateEnclosingElementRuntimeData.get
lazy val choice = new AltCompParser(context.runtimeData, parsers)
lazy val rep = new RepUnboundedParser(OccursCountKind.Parsed, choice, erd) // Unordered is a RepUnbounded of Choice.
Console.out.println(rep)
lazy val parser: DaffodilParser = new UnorderedSequenceCombinatorParser(s.termRuntimeData, rep, sortOrder, scalarMembers)
}
class UnorderedSequenceCombinatorParser(rd: RuntimeData, bodyParser: Parser,
sortOrder: Seq[(NamedQName, NS)],
scalarMembers: Seq[(NamedQName, NS)])
extends ParserObject(rd) {
override def nom = "UnorderedSequence"
override lazy val childProcessors = Seq(bodyParser)
...
def parse(start: PState): Unit = {
start.mpstate.groupIndexStack.push(1L) // one-based indexing
bodyParser.parse1(start) // Body parser here should be the RepUnbounded of Choice
val elt = start.thisElement
val complex = elt.asComplex
// Removes 'choiceElement'
//val childrenDetached = complex.childNodes.flatMap(n => n.contents.flatMap(c => c.contents))
val childrenDetached = complex.childNodes
checkScalarsOccurExactlyOnce(childrenDetached, start)
// dropDefaulted(currentElemAfter, end)
checkOccurrence(elt, childrenDetached, start)
// Sort so that the contents are in the expected order.
sort(complex, childrenDetached, start)
start.mpstate.groupIndexStack.pop()
start.mpstate.moveOverOneGroupIndexOnly()
()
}
}
This allows for the same functionality as the hack of mutating the XML of an unordered sequence into a sequence of choice.
If you look at the following test, just simply using a combination of the appropriate parsers is not enough.
<tdml:defineSchema name="simple_min_max_occurs">
<dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
lengthKind="implicit" />
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="unordered">
<xs:element name="e1" type="ex:stNumStr"
dfdl:lengthKind="explicit" dfdl:length="2" minOccurs="1"
maxOccurs="2" dfdl:occursCountKind="parsed">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="e2" type="ex:stHexStr"
dfdl:lengthKind="explicit" dfdl:length="2">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="e3" type="xs:string" dfdl:lengthKind="explicit"
dfdl:length="2">
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ dfdl:checkConstraints(.) }"
message="Assertion failed for dfdl:checkConstraints(.)" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:simpleType name="stHexStr">
<xs:restriction base="xs:string">
<xs:pattern value="[a-fA-F0-9]+"></xs:pattern>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="stNumStr">
<xs:restriction base="xs:string">
<xs:pattern value="[0-9]+"></xs:pattern>
</xs:restriction>
</xs:simpleType>
</tdml:defineSchema>
<tdml:parserTestCase name="test_simple_min_max_occurs"
model="simple_min_max_occurs" description="Section 14 Sequence group with left over data - DFDL-14-001R"
root="USG_01" validation="on">
<tdml:document><![CDATA[veba0022]]></tdml:document>
<tdml:infoset>
<tdml:dfdlInfoset>
<USG_01>
<e1>00</e1>
<e1>22</e1>
<e2>ba</e2>
<e3>ve</e3>
</USG_01>
</tdml:dfdlInfoset>
</tdml:infoset>
</tdml:parserTestCase>
Why? Because 'parsers' in the code above looks like the following:
List(<Array><RepUnbounded name='e1' n='-1'><Element name='e1'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e1 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element></RepUnbounded></Array>, <Element name='e2'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e2 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element>, <Element name='e3'><AlignmentFillParser/><seq><MandatoryTextAlignmentParser/><SpecifiedLengthExplicitParser><e3 parser='StringOfSpecifiedLengthParser' /></SpecifiedLengthExplicitParser></seq><AssertExpressionEvaluationParser/></Element>)
The problem here is that it wants to do a RepUnbounded for e1. This firmly expects to encounter e1 1 to 2 times. Because we actually encounter 've' first this RepUnbounded for e1 fails but exits with a failure suppressed. This then basically tells the AltComp that the choice was successful and causes the parse to end prematurely with the following error:
edu.illinois.ncsa.daffodil.tdml.TDMLException: Runtime Schema Definition Error: {http://example.com}e2 is scalar but occurred 0 times.
Path: http://example.com
Offending children:
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
Parse Error: RepUnbounded - No forward progress at byte 0. Attempt to parse ex:USG_01 succeeded but consumed no data.
Please re-examine your schema to correct this infinite loop.
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
Validation Error: e1 occurred '0' times when it was expected to be a minimum of '1' and a maximum of '2' times.
Schema context: ex:USG_01 Location line 26 column 10 in file:/tmp/simple_min_max_occurs6683421060854835730.dfdl.xsd
Data location was preceding byte 0 limit(bytes) 8
I believe the fix for this would be to alter the grammar in the case of unordered sequences. Meaning that at the point where we decide whether the termContentBody of the LocalElement is
an enclosedElement or recurrance, we need to make it an enclosedElement if it's the immediate child of an unordered sequence.
Thoughts?
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Thursday, November 30, 2017 4:18:35 PM
To: Taylor Wise; dev@daffodil.apache.org; Mike Beckerle
Subject: Re: Unordered Sequence and Choice Branch Violations
I think we should just remove the choiceElement entirely. A new
UnorderedSequenceCombinatorParser should be able to handle all the logic
that is currently handled by converting unordered sequences to a
sequence of choices (which looks to be done in the lazy val unorderedSeq
in SequenceGroup.scala). It does seem like this expressions accessing
other branches of an unordered sequences should be an SDE.
On 11/30/2017 03:39 PM, Taylor Wise wrote:
> By this same token then, any assertions/expressions that navigate to a different
> branch of the unordered sequence/choice are invalid.
>
>
> "When processing a choice group the parser validates any contained path
> expressions. If a path expression contained inside a choice branch refers to any
> other branch of the choice, then it is a schema definition error. Note that this
> rule handles nested choices also. A path that navigates outward from an inner
> choice to another alternative of an outer choice is violating this rule with
> respect to the outer choice."
>
>
> I'm currently running into an issue where there's an existing test that has an
> expression in the initiator of an element in an unordered sequence. The
> expression refers to a prior element. Because of the spec, this should SDE.
>
>
> However, it's currently failing stating that the element does not exist when it
> should have already been added. This is leading me to believe that there is
> either something funky going on with the expression steps due to the additional
> choiceElement or that somehow the element (e3) isn't being kept around in the
> infoset (unlikely as tests without expressions run fine).
>
>
> The existing test:
>
>
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="unordered"
> dfdl:separator=",">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
> model="simple_invalid_path_to_branch" description="Section 14 Sequence group
> with left over data - DFDL-14-001R"
> root="USG_01" roundTrip="false">
> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
> <tdml:errors>
> <tdml:error>Schema Definition Error</tdml:error>
> <tdml:error>Expression</tdml:error>
> <tdml:error>../e3</tdml:error>
> <tdml:error>navigates to another branch</tdml:error>
> <tdml:error>e3</tdml:error>
> </tdml:errors>
> </tdml:parserTestCase>
> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message
> "../e3" in any of the actual diagnostic messages:
> Runtime Schema Definition Error: Expression Evaluation Error: Child element
> {http://example.com}e3 does not exist.
> Schema context: e4 Location line 199 column 22 in
> file:/tmp/simple_invalid_path_to_branch1343627752058513822.dfdl.xsd
> Data location was preceding byte 12 limit(bytes) 19
>
> The source-to-source transformation:
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="ordered"
> dfdl:separator=",">
> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
> occursCountKind="parsed">
> <xs:complexType>
> <xs:choice dfdl:choiceLengthKind="implicit">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:choice>
> </xs:complexType>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> In this case, e3 should definitely exist by the time the choice alternative for
> e4 is run. Stepping through, I can see each alternative succeed for each element
> until it tries e4.
> At which time, it doesn't appear that e3 is in the infoset (though it should
> be). What appears to be happening is that each time an alternative succeeds,
> a choiceElement (containing the parsed element) is added to USG_01. So I think
> what's happening is that these choiceElement's are interfering with the
> expression evaluations.
>
> You end up with something like this in the Infoset:
> <USG_01>
> <choiceElement><e2>2</e2></choiceElement>
> <choiceElement><e1>1</e1></choiceElement>
> <choiceElement><e1>4</e1></choiceElement>
> <choiceElement><e3>3</e3></choiceElement>
> <choiceElement> ....Let's look for the Initiator from e3... </choiceElement>
> </USG_01>
>
> The expression is expecting to be able to go UP to USG_01 and ask for a child
> named e3. But, it would appear that it's going UP to its choiceElement and
> asking for e3.
> Which does not exist.
>
> Which brings me to two possible solutions:
>
> 1. Somehow detect that we're running an expression from within a choice and
> 'hide/skip' choiceElements. Likely meaning we'll have to prepend an extra
> ../ somewhere. This adds some unnecessary complexity to the expression
> code. Which I kind of want to avoid.
> 2. Remove the choiceElement entirely.
>
>
>
>
>
>
> --------------------------------------------------------------------------------
> *From:* Steve Lawrence <sl...@apache.org>
> *Sent:* Friday, November 17, 2017 1:37:46 PM
> *To:* dev@daffodil.apache.org; Taylor Wise; Mike Beckerle
> *Subject:* Re: Unordered Sequence and Choice Branch Violations
> On 11/17/2017 01:13 PM, Taylor Wise wrote:
>> While executing some existing tests I've come across a case that I need guidance on.
>>
>>
>> Given the following test case:
>>
>> <tdml:defineSchema name="simple_invalid_path_to_branch">
>> <dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
>> lengthKind="delimited" />
>> <xs:element name="USG_01">
>> <xs:complexType>
>> <xs:sequence dfdl:sequenceKind="unordered"
>> dfdl:separator=",">
>>
>> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="a">
>> </xs:element>
>>
>> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="b">
>> </xs:element>
>>
>> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="c">
>> </xs:element>
>>
>> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
>> dfdl:inputValueCalc="{ ../ex:e3 }">
>> </xs:element>
>>
>> </xs:sequence>
>> </xs:complexType>
>> </xs:element>
>>
>>
>> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
>> model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
>> root="USG_01" roundTrip="false">
>> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
>> <tdml:errors>
>> <tdml:error>Schema Definition Error</tdml:error>
>> <tdml:error>Expression</tdml:error>
>> <tdml:error>../e3</tdml:error>
>> <tdml:error>navigates to another branch</tdml:error>
>> <tdml:error>e3</tdml:error>
>> </tdml:errors>
>> </tdml:parserTestCase>
>>
>> The result of the test is the following:
>>
>> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
>> Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
>> Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
>>
>> Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
>>
>> Something like:
>>
>> <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
>>
>> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
>> dfdl:occursCountKind="parsed">
>>
>> <xs:complexType>
>>
>> <xs:choice dfdl:choiceLengthKind="implicit">
>>
>> <xs:element name="e1" type="xs:string" dfdl:initiator="a" />
>>
>> <xs:element name="e2" type="xs:int" dfdl:initiator="b" />
>>
>> <xs:element name="e3" type="xs:string" dfdl:initiator="c" />
>>
>> <xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
>>
>> </xs:choice>
>>
>> </xs:complexType>
>>
>> </xs:element>
>>
>> </xs:sequence>
>>
>> So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
>> The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
>>
>> Offending code:
>> final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
>> val branchesOk = groupMembersNoRefs map { branch =>
>> if (!branch.isRepresented) {
>> schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
>> false
>> } else true
>> }
>> assuming(branchesOk.forall { x => x })
>> }.value
>>
>>
>> About IVC from the spec:
>> It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
>> It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
>> It is not possible to place this property in scope on a dfdl:format annotation.
>> This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
>> If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
>> If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
>> If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
>>
>
> Seems to me like this shouldn't be allowed to me. I don't think
> inputValueCalc elements even consume any data, so it doesn't make sense
> to be in an unordered sequence. Otherwise you could parse
> inputValueCalc's forever as part of that sequence.
>
> I'll also add that if we add a new UnordredSequenceCombinator, I would
> think we could get rid of the code that creates a fake sequence of
> choices. That always seemed a bit of a hack and made things awkward, for
> example the "choiceElement" showing up in the path.
>
Re: Unordered Sequence and Choice Branch Violations
Posted by Steve Lawrence <sl...@apache.org>.
I think we should just remove the choiceElement entirely. A new
UnorderedSequenceCombinatorParser should be able to handle all the logic
that is currently handled by converting unordered sequences to a
sequence of choices (which looks to be done in the lazy val unorderedSeq
in SequenceGroup.scala). It does seem like this expressions accessing
other branches of an unordered sequences should be an SDE.
On 11/30/2017 03:39 PM, Taylor Wise wrote:
> By this same token then, any assertions/expressions that navigate to a different
> branch of the unordered sequence/choice are invalid.
>
>
> "When processing a choice group the parser validates any contained path
> expressions. If a path expression contained inside a choice branch refers to any
> other branch of the choice, then it is a schema definition error. Note that this
> rule handles nested choices also. A path that navigates outward from an inner
> choice to another alternative of an outer choice is violating this rule with
> respect to the outer choice."
>
>
> I'm currently running into an issue where there's an existing test that has an
> expression in the initiator of an element in an unordered sequence. The
> expression refers to a prior element. Because of the spec, this should SDE.
>
>
> However, it's currently failing stating that the element does not exist when it
> should have already been added. This is leading me to believe that there is
> either something funky going on with the expression steps due to the additional
> choiceElement or that somehow the element (e3) isn't being kept around in the
> infoset (unlikely as tests without expressions run fine).
>
>
> The existing test:
>
>
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="unordered"
> dfdl:separator=",">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
> model="simple_invalid_path_to_branch" description="Section 14 Sequence group
> with left over data - DFDL-14-001R"
> root="USG_01" roundTrip="false">
> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
> <tdml:errors>
> <tdml:error>Schema Definition Error</tdml:error>
> <tdml:error>Expression</tdml:error>
> <tdml:error>../e3</tdml:error>
> <tdml:error>navigates to another branch</tdml:error>
> <tdml:error>e3</tdml:error>
> </tdml:errors>
> </tdml:parserTestCase>
> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message
> "../e3" in any of the actual diagnostic messages:
> Runtime Schema Definition Error: Expression Evaluation Error: Child element
> {http://example.com}e3 does not exist.
> Schema context: e4 Location line 199 column 22 in
> file:/tmp/simple_invalid_path_to_branch1343627752058513822.dfdl.xsd
> Data location was preceding byte 12 limit(bytes) 19
>
> The source-to-source transformation:
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="ordered"
> dfdl:separator=",">
> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
> occursCountKind="parsed">
> <xs:complexType>
> <xs:choice dfdl:choiceLengthKind="implicit">
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a"/>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b"/>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c"/>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:initiator="{ ../ex:e3 }"/>
> </xs:choice>
> </xs:complexType>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
> In this case, e3 should definitely exist by the time the choice alternative for
> e4 is run. Stepping through, I can see each alternative succeed for each element
> until it tries e4.
> At which time, it doesn't appear that e3 is in the infoset (though it should
> be). What appears to be happening is that each time an alternative succeeds,
> a choiceElement (containing the parsed element) is added to USG_01. So I think
> what's happening is that these choiceElement's are interfering with the
> expression evaluations.
>
> You end up with something like this in the Infoset:
> <USG_01>
> <choiceElement><e2>2</e2></choiceElement>
> <choiceElement><e1>1</e1></choiceElement>
> <choiceElement><e1>4</e1></choiceElement>
> <choiceElement><e3>3</e3></choiceElement>
> <choiceElement> ....Let's look for the Initiator from e3... </choiceElement>
> </USG_01>
>
> The expression is expecting to be able to go UP to USG_01 and ask for a child
> named e3. But, it would appear that it's going UP to its choiceElement and
> asking for e3.
> Which does not exist.
>
> Which brings me to two possible solutions:
>
> 1. Somehow detect that we're running an expression from within a choice and
> 'hide/skip' choiceElements. Likely meaning we'll have to prepend an extra
> ../ somewhere. This adds some unnecessary complexity to the expression
> code. Which I kind of want to avoid.
> 2. Remove the choiceElement entirely.
>
>
>
>
>
>
> --------------------------------------------------------------------------------
> *From:* Steve Lawrence <sl...@apache.org>
> *Sent:* Friday, November 17, 2017 1:37:46 PM
> *To:* dev@daffodil.apache.org; Taylor Wise; Mike Beckerle
> *Subject:* Re: Unordered Sequence and Choice Branch Violations
> On 11/17/2017 01:13 PM, Taylor Wise wrote:
>> While executing some existing tests I've come across a case that I need guidance on.
>>
>>
>> Given the following test case:
>>
>> <tdml:defineSchema name="simple_invalid_path_to_branch">
>> <dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
>> lengthKind="delimited" />
>> <xs:element name="USG_01">
>> <xs:complexType>
>> <xs:sequence dfdl:sequenceKind="unordered"
>> dfdl:separator=",">
>>
>> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="a">
>> </xs:element>
>>
>> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="b">
>> </xs:element>
>>
>> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
>> dfdl:initiator="c">
>> </xs:element>
>>
>> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
>> dfdl:inputValueCalc="{ ../ex:e3 }">
>> </xs:element>
>>
>> </xs:sequence>
>> </xs:complexType>
>> </xs:element>
>>
>>
>> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
>> model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
>> root="USG_01" roundTrip="false">
>> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
>> <tdml:errors>
>> <tdml:error>Schema Definition Error</tdml:error>
>> <tdml:error>Expression</tdml:error>
>> <tdml:error>../e3</tdml:error>
>> <tdml:error>navigates to another branch</tdml:error>
>> <tdml:error>e3</tdml:error>
>> </tdml:errors>
>> </tdml:parserTestCase>
>>
>> The result of the test is the following:
>>
>> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
>> Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
>> Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
>>
>> Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
>>
>> Something like:
>>
>> <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
>>
>> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
>> dfdl:occursCountKind="parsed">
>>
>> <xs:complexType>
>>
>> <xs:choice dfdl:choiceLengthKind="implicit">
>>
>> <xs:element name="e1" type="xs:string" dfdl:initiator="a" />
>>
>> <xs:element name="e2" type="xs:int" dfdl:initiator="b" />
>>
>> <xs:element name="e3" type="xs:string" dfdl:initiator="c" />
>>
>> <xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
>>
>> </xs:choice>
>>
>> </xs:complexType>
>>
>> </xs:element>
>>
>> </xs:sequence>
>>
>> So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
>> The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
>>
>> Offending code:
>> final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
>> val branchesOk = groupMembersNoRefs map { branch =>
>> if (!branch.isRepresented) {
>> schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
>> false
>> } else true
>> }
>> assuming(branchesOk.forall { x => x })
>> }.value
>>
>>
>> About IVC from the spec:
>> It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
>> It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
>> It is not possible to place this property in scope on a dfdl:format annotation.
>> This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
>> If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
>> If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
>> If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
>>
>
> Seems to me like this shouldn't be allowed to me. I don't think
> inputValueCalc elements even consume any data, so it doesn't make sense
> to be in an unordered sequence. Otherwise you could parse
> inputValueCalc's forever as part of that sequence.
>
> I'll also add that if we add a new UnordredSequenceCombinator, I would
> think we could get rid of the code that creates a fake sequence of
> choices. That always seemed a bit of a hack and made things awkward, for
> example the "choiceElement" showing up in the path.
>
Re: Unordered Sequence and Choice Branch Violations
Posted by Taylor Wise <tw...@tresys.com>.
By this same token then, any assertions/expressions that navigate to a different branch of the unordered sequence/choice are invalid.
"When processing a choice group the parser validates any contained path expressions. If a path expression contained inside a choice branch refers to any other branch of the choice, then it is a schema definition error. Note that this rule handles nested choices also. A path that navigates outward from an inner choice to another alternative of an outer choice is violating this rule with respect to the outer choice."
I'm currently running into an issue where there's an existing test that has an expression in the initiator of an element in an unordered sequence. The expression refers to a prior element. Because of the spec, this should SDE.
However, it's currently failing stating that the element does not exist when it should have already been added. This is leading me to believe that there is either something funky going on with the expression steps due to the additional choiceElement or that somehow the element (e3) isn't being kept around in the infoset (unlikely as tests without expressions run fine).
The existing test:
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="unordered"
dfdl:separator=",">
<xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="a"/>
<xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="b"/>
<xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="c"/>
<xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
dfdl:initiator="{ ../ex:e3 }"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<tdml:parserTestCase name="test_simple_invalid_path_to_branch"
model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
root="USG_01" roundTrip="false">
<tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
<tdml:errors>
<tdml:error>Schema Definition Error</tdml:error>
<tdml:error>Expression</tdml:error>
<tdml:error>../e3</tdml:error>
<tdml:error>navigates to another branch</tdml:error>
<tdml:error>e3</tdml:error>
</tdml:errors>
</tdml:parserTestCase>
edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "../e3" in any of the actual diagnostic messages:
Runtime Schema Definition Error: Expression Evaluation Error: Child element {http://example.com}e3 does not exist.
Schema context: e4 Location line 199 column 22 in file:/tmp/simple_invalid_path_to_branch1343627752058513822.dfdl.xsd
Data location was preceding byte 12 limit(bytes) 19
The source-to-source transformation:
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="ordered"
dfdl:separator=",">
<xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
occursCountKind="parsed">
<xs:complexType>
<xs:choice dfdl:choiceLengthKind="implicit">
<xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="a"/>
<xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="b"/>
<xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="c"/>
<xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
dfdl:initiator="{ ../ex:e3 }"/>
</xs:choice>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
In this case, e3 should definitely exist by the time the choice alternative for e4 is run. Stepping through, I can see each alternative succeed for each element until it tries e4.
At which time, it doesn't appear that e3 is in the infoset (though it should be). What appears to be happening is that each time an alternative succeeds,
a choiceElement (containing the parsed element) is added to USG_01. So I think what's happening is that these choiceElement's are interfering with the expression evaluations.
You end up with something like this in the Infoset:
<USG_01>
<choiceElement><e2>2</e2></choiceElement>
<choiceElement><e1>1</e1></choiceElement>
<choiceElement><e1>4</e1></choiceElement>
<choiceElement><e3>3</e3></choiceElement>
<choiceElement> ....Let's look for the Initiator from e3... </choiceElement>
</USG_01>
The expression is expecting to be able to go UP to USG_01 and ask for a child named e3. But, it would appear that it's going UP to its choiceElement and asking for e3.
Which does not exist.
Which brings me to two possible solutions:
1. Somehow detect that we're running an expression from within a choice and 'hide/skip' choiceElements. Likely meaning we'll have to prepend an extra ../ somewhere. This adds some unnecessary complexity to the expression code. Which I kind of want to avoid.
2. Remove the choiceElement entirely.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Friday, November 17, 2017 1:37:46 PM
To: dev@daffodil.apache.org; Taylor Wise; Mike Beckerle
Subject: Re: Unordered Sequence and Choice Branch Violations
On 11/17/2017 01:13 PM, Taylor Wise wrote:
> While executing some existing tests I've come across a case that I need guidance on.
>
>
> Given the following test case:
>
> <tdml:defineSchema name="simple_invalid_path_to_branch">
> <dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
> lengthKind="delimited" />
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="unordered"
> dfdl:separator=",">
>
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a">
> </xs:element>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b">
> </xs:element>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c">
> </xs:element>
>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:inputValueCalc="{ ../ex:e3 }">
> </xs:element>
>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
>
> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
> model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
> root="USG_01" roundTrip="false">
> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
> <tdml:errors>
> <tdml:error>Schema Definition Error</tdml:error>
> <tdml:error>Expression</tdml:error>
> <tdml:error>../e3</tdml:error>
> <tdml:error>navigates to another branch</tdml:error>
> <tdml:error>e3</tdml:error>
> </tdml:errors>
> </tdml:parserTestCase>
>
> The result of the test is the following:
>
> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
> Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
> Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
>
> Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
>
> Something like:
>
> <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
>
> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
> dfdl:occursCountKind="parsed">
>
> <xs:complexType>
>
> <xs:choice dfdl:choiceLengthKind="implicit">
>
> <xs:element name="e1" type="xs:string" dfdl:initiator="a" />
>
> <xs:element name="e2" type="xs:int" dfdl:initiator="b" />
>
> <xs:element name="e3" type="xs:string" dfdl:initiator="c" />
>
> <xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
>
> </xs:choice>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
> The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
>
> Offending code:
> final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
> val branchesOk = groupMembersNoRefs map { branch =>
> if (!branch.isRepresented) {
> schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
> false
> } else true
> }
> assuming(branchesOk.forall { x => x })
> }.value
>
>
> About IVC from the spec:
> It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
> It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
> It is not possible to place this property in scope on a dfdl:format annotation.
> This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
> If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
> If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
> If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
>
Seems to me like this shouldn't be allowed to me. I don't think
inputValueCalc elements even consume any data, so it doesn't make sense
to be in an unordered sequence. Otherwise you could parse
inputValueCalc's forever as part of that sequence.
I'll also add that if we add a new UnordredSequenceCombinator, I would
think we could get rid of the code that creates a fake sequence of
choices. That always seemed a bit of a hack and made things awkward, for
example the "choiceElement" showing up in the path.
Re: Unordered Sequence and Choice Branch Violations
Posted by Steve Lawrence <sl...@apache.org>.
On 11/17/2017 01:13 PM, Taylor Wise wrote:
> While executing some existing tests I've come across a case that I need guidance on.
>
>
> Given the following test case:
>
> <tdml:defineSchema name="simple_invalid_path_to_branch">
> <dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
> lengthKind="delimited" />
> <xs:element name="USG_01">
> <xs:complexType>
> <xs:sequence dfdl:sequenceKind="unordered"
> dfdl:separator=",">
>
> <xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="a">
> </xs:element>
>
> <xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="b">
> </xs:element>
>
> <xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
> dfdl:initiator="c">
> </xs:element>
>
> <xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
> dfdl:inputValueCalc="{ ../ex:e3 }">
> </xs:element>
>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
>
>
> <tdml:parserTestCase name="test_simple_invalid_path_to_branch"
> model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
> root="USG_01" roundTrip="false">
> <tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
> <tdml:errors>
> <tdml:error>Schema Definition Error</tdml:error>
> <tdml:error>Expression</tdml:error>
> <tdml:error>../e3</tdml:error>
> <tdml:error>navigates to another branch</tdml:error>
> <tdml:error>e3</tdml:error>
> </tdml:errors>
> </tdml:parserTestCase>
>
> The result of the test is the following:
>
> edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
> Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
> Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
>
> Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
>
> Something like:
>
> <xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
>
> <xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
> dfdl:occursCountKind="parsed">
>
> <xs:complexType>
>
> <xs:choice dfdl:choiceLengthKind="implicit">
>
> <xs:element name="e1" type="xs:string" dfdl:initiator="a" />
>
> <xs:element name="e2" type="xs:int" dfdl:initiator="b" />
>
> <xs:element name="e3" type="xs:string" dfdl:initiator="c" />
>
> <xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
>
> </xs:choice>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
> The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
>
> Offending code:
> final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
> val branchesOk = groupMembersNoRefs map { branch =>
> if (!branch.isRepresented) {
> schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
> false
> } else true
> }
> assuming(branchesOk.forall { x => x })
> }.value
>
>
> About IVC from the spec:
> It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
> It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
> It is not possible to place this property in scope on a dfdl:format annotation.
> This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
> If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
> If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
> If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.
>
Seems to me like this shouldn't be allowed to me. I don't think
inputValueCalc elements even consume any data, so it doesn't make sense
to be in an unordered sequence. Otherwise you could parse
inputValueCalc's forever as part of that sequence.
I'll also add that if we add a new UnordredSequenceCombinator, I would
think we could get rid of the code that creates a fake sequence of
choices. That always seemed a bit of a hack and made things awkward, for
example the "choiceElement" showing up in the path.
Re: Unordered Sequence and Choice Branch Violations
Posted by Mike Beckerle <mb...@tresys.com>.
Hmmm. So an IVC element within an unordered sequence? When would it get evaluated?
I think we should prevent this. The whole notion of unordered has to do with positions of things in the representation. A non-represented element has no business being in there.
So I think the same restriction should apply as for choice branches - no IVC allowed.
________________________________
From: Taylor Wise
Sent: Friday, November 17, 2017 1:13:12 PM
To: Mike Beckerle; Steve Lawrence; dev@daffodil.apache.org
Subject: Unordered Sequence and Choice Branch Violations
While executing some existing tests I've come across a case that I need guidance on.
Given the following test case:
<tdml:defineSchema name="simple_invalid_path_to_branch">
<dfdl:format ref="ex:daffodilTest1" lengthUnits="characters"
lengthKind="delimited" />
<xs:element name="USG_01">
<xs:complexType>
<xs:sequence dfdl:sequenceKind="unordered"
dfdl:separator=",">
<xs:element name="e1" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="a">
</xs:element>
<xs:element name="e2" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="b">
</xs:element>
<xs:element name="e3" type="xs:int" dfdl:lengthKind="delimited"
dfdl:initiator="c">
</xs:element>
<xs:element name="e4" type="xs:string" dfdl:lengthKind="delimited"
dfdl:inputValueCalc="{ ../ex:e3 }">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<tdml:parserTestCase name="test_simple_invalid_path_to_branch"
model="simple_invalid_path_to_branch" description="Section 14 Sequence group with left over data - DFDL-14-001R"
root="USG_01" roundTrip="false">
<tdml:document><![CDATA[b2,a1,a4,c3,ruh-oh!]]></tdml:document>
<tdml:errors>
<tdml:error>Schema Definition Error</tdml:error>
<tdml:error>Expression</tdml:error>
<tdml:error>../e3</tdml:error>
<tdml:error>navigates to another branch</tdml:error>
<tdml:error>e3</tdml:error>
</tdml:errors>
</tdml:parserTestCase>
The result of the test is the following:
edu.illinois.ncsa.daffodil.tdml.TDMLException: Did not find diagnostic message "Expression" in any of the actual diagnostic messages:
Schema Definition Error: Branch of choice ex:USG_01::LocalComplexTypeDef::sequence::choiceElement::LocalComplexTypeDef::choice::e4 cannot have the dfdl:inputValueCalc property.
Schema context: choice Location line 184 in file:/tmp/simple_invalid_path_to_branch4208585747499063196.dfdl.xsd
Now, in the test you'll notice that there's no choice listed in the schema. But because it's an unordered sequence, it gets transformed into a sequence of choice.
Something like:
<xs:sequence dfdl:sequenceKind="ordered" dfdl:separator=",">
<xs:element name="choiceElement" minOccurs="0" maxOccurs="unbounded"
dfdl:occursCountKind="parsed">
<xs:complexType>
<xs:choice dfdl:choiceLengthKind="implicit">
<xs:element name="e1" type="xs:string" dfdl:initiator="a" />
<xs:element name="e2" type="xs:int" dfdl:initiator="b" />
<xs:element name="e3" type="xs:string" dfdl:initiator="c" />
<xs:element name="e4" type="xs:string" dfdl:inputValueCalc="{ ../ex:e3 }" />
</xs:choice>
</xs:complexType>
</xs:element>
</xs:sequence>
So does this now mean that this schema has to obey all the rules of choice? That's not immediately obvious.
The spec states that IVC is "not allowed to appear on a local element or element reference that is the root of a choice branch". e4 is now the root of a choice branch.
Offending code:
final def branchesAreNotIVCElements = LV('branchesAreNotIVCElements) {
val branchesOk = groupMembersNoRefs map { branch =>
if (!branch.isRepresented) {
schemaDefinitionErrorButContinue("Branch of choice %s cannot have the dfdl:inputValueCalc property.".format(branch.path))
false
} else true
}
assuming(branchesOk.forall { x => x })
}.value
About IVC from the spec:
It is a schema definition error if this property is specified on an element which has an XSDL fixed or default property.
It is a schema definition error if dfdl:inputValueCalc and dfdl:outputValueCalc are specified on the same element.
It is not possible to place this property in scope on a dfdl:format annotation.
This property is not allowed to appear on a local element or element reference that is the root of a choice branch.
If this property appears on an element declaration or element reference schema component, the appearance of any other DFDL properties on that component is a schema definition error.
If this property appears on an element reference, then DFDL properties expressed on the referenced global element declaration or its type are ignored.
If this property appears on an element declaration, then DFDL properties expressed on its type are ignored.