You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Mike Beckerle <mb...@tresys.com> on 2017/11/10 13:21:03 UTC
Stack overflows, UnorderedSequences, DSOM
So I've had some stack-overflows while debugging of late.
This has lead to thinking about the reasons why. There are some principles of operation for the code-base of Daffodil that are not articulated, but that need to be followed to stay out of trouble.
I am planning to add package.scala files describing these to the code base.
For DSOM one is this:
* DSOM components should be selected and constructed without reference to DFDL annotations, (including DFDL properties) carried on any schema component.
This avoids a chicken/egg situation. The DSOM graph is needed in order to implement DFDL's property scoping rules; so you can't invoke those mechanisms in order to build the DSOM graph itself, or you will very likely end up with a circularity/stack-overflow.
You can; however, look at the XML and its attributes. E.g., the XSD ref property, which is a QName for a global element/type/group. There are also some DFDL non-properties - cannot be put into scope - that are needed at the time DSOM objects are being constructed. One example is the dfdl:hiddenGroupRef "property" on xs:sequence XML elements. In general it is acceptable to look at, and even resolve global QName references at DSOM graph construction time.
Coming back to the issue of unordered sequences...
The above suggests a design flaw in DSOM currently, which is the UnorderedSequence class. In order to decide whether to construct this class or just a regular Sequence class instance, you have to examine the dfdl:sequenceKind property, which can be scoped.
Hence, I claim, the UnorderedSequence class should not exist in current form, but an isUnordered attribute on Sequence should provide the information about whether the class is ordered or not, and that is determined only *after* the DSOM graph is constructed. Once the DSOM Sequence class has been constructed, then regular (usually lazy val) class members on Sequence can construct any helper objects that are needed for dealing with the complexity of compilation for unordered sequences.
Comments?
Re: Stack overflows, UnorderedSequences, DSOM
Posted by Taylor Wise <tw...@tresys.com>.
Are you saying then that we shouldn't look at sequenceKind variable, but rather grab the sequenceKind directly from the XML?
So rather than:
lazy val isOrdered = sequenceKind match {
case SequenceKind.Ordered => true
case SequenceKind.Unordered => false
}
Do:
lazy val isOrdered = xml.attributes.get("sequenceKind") match {
case None => true // Ordered
case Some(order) => SequenceKind(order, this) match { // enum construction unnecessary
case SequenceKind.Ordered => true
case SequenceKind.Unordered => false
}
}
________________________________
From: Mike Beckerle
Sent: Friday, November 10, 2017 8:21:03 AM
To: dev@daffodil.apache.org; Taylor Wise
Subject: Stack overflows, UnorderedSequences, DSOM
So I've had some stack-overflows while debugging of late.
This has lead to thinking about the reasons why. There are some principles of operation for the code-base of Daffodil that are not articulated, but that need to be followed to stay out of trouble.
I am planning to add package.scala files describing these to the code base.
For DSOM one is this:
* DSOM components should be selected and constructed without reference to DFDL annotations, (including DFDL properties) carried on any schema component.
This avoids a chicken/egg situation. The DSOM graph is needed in order to implement DFDL's property scoping rules; so you can't invoke those mechanisms in order to build the DSOM graph itself, or you will very likely end up with a circularity/stack-overflow.
You can; however, look at the XML and its attributes. E.g., the XSD ref property, which is a QName for a global element/type/group. There are also some DFDL non-properties - cannot be put into scope - that are needed at the time DSOM objects are being constructed. One example is the dfdl:hiddenGroupRef "property" on xs:sequence XML elements. In general it is acceptable to look at, and even resolve global QName references at DSOM graph construction time.
Coming back to the issue of unordered sequences...
The above suggests a design flaw in DSOM currently, which is the UnorderedSequence class. In order to decide whether to construct this class or just a regular Sequence class instance, you have to examine the dfdl:sequenceKind property, which can be scoped.
Hence, I claim, the UnorderedSequence class should not exist in current form, but an isUnordered attribute on Sequence should provide the information about whether the class is ordered or not, and that is determined only *after* the DSOM graph is constructed. Once the DSOM Sequence class has been constructed, then regular (usually lazy val) class members on Sequence can construct any helper objects that are needed for dealing with the complexity of compilation for unordered sequences.
Comments?
Re: Stack overflows, UnorderedSequences, DSOM
Posted by Mike Beckerle <mb...@tresys.com>.
So this issue of when DSOM is "done" is key. DSOM is done when all the objects that are needed to resolve DFDL annotations have been constructed.
I would say this is when all DSOM objects corresponding to "pieces of XML" in the schema exist, and when the formatAnnotation for each AnnotatedSchemaComponent exists. At that point you can reference properties and other annotations like statements all you want, and SDEs won't end up in circular stack overflows due to the need for a DSOM object.
Note that top level decls/defs might not have any DSOM object created for them if they are not referenced.
But this is ALL before anything invokes any method/member of any of the Grammar mixins. So the code you cite below is (or should be) after the DSOM graph is fully constructed.
It is exactly this use of dfdl:sequenceKind to decide what DSOM object to create that is, I believe, a mistake in DSOM. The selection of either a Sequence or UnorderedSequence DSOM object in the DSOM graph is depending on dfdl:sequenceKind, which is a scoped DFDL property, and scope resolution depends on the DSOM graph. Resolution of the dfdl:sequenceKind method has to start from the correct start of the chain.
Consider this:
---File 1----
<dfdl:format sequenceKind="unordered".../>
... elsewhere in that file....
<group ref="mySeqGr"/>
----File 2----
<group name="mySeqGr">
<sequence> <!-- will be ordered or not depending on group refs -->
...
</sequence>
</group>
So to determine ordered or unordered, we have to look back at the group reference and start the scan for properties there. This property might even come from the default format of the schema file containing the group reference, so we really need the proper DSOM structures in place before we ask for the dfdl:sequenceKind.
I bet this wouldn't work today because Daffodil would miss the dfdl:sequenceKind found by way of the group ref. Maybe it would work, but I wouldn't bet on it.
The right design is to construct a Sequence every time, and construct the children DSOM objects of the Sequence every time. We have the UnorderedSequence object become, if needed, just a helper object used by the Sequence. Honestly a helper object may not be needed. The Sequence method code can just evaluate some lazy vals to decide whether to put down SeqComp or UnorderedSequenceCombinator, and compute the parameters they need from the DSOM objects for the contained children.
One way to do this with some uniformity is to construct a Sequence every time as part of the DSOM tree, and then construct the DSOM objects for that Sequence's children. After that and when DSOM is fully constructed, then when that Sequence is asked for some member/method then if that Sequence determines it is unordered, to then construct a UnorderedSequence, (passing in the DSOM objects for the children to the constructor). The original Sequence would then delegate everything to the UnorderedSequence object. The consistent thing to do would then be to have dsom.Sequence construct either an OrderedSequence or an UnorderedSequence, both of which implement the same interface. The Sequence then decides which one to create, and delegates everything to that object.
It's using a delegation pattern instead of an inheritance pattern because we want to decide which class to create later, when we have more information.
In that case, the OrderedSequence and UnorderedSequence objects might be DSOM objects out of convenience. But they're not ordinary DSOM objects because they do not recursively parse their own children into DSOM objects. The original Sequence object would be a DSOM object, but really would just be a thin proxy delegating everything except child construction to the other objects.
________________________________
From: Taylor Wise
Sent: Friday, November 10, 2017 10:07:25 AM
To: Mike Beckerle; dev@daffodil.apache.org
Subject: Re: Stack overflows, UnorderedSequences, DSOM
Also, at what point is it safe to construct these helper objects? When is DSOM done?
For instance, if you go to SequenceGrammarMixin you'll see that we have a bunch of lazy evaluations concerning orderedSequenceContent and unorderedSequenceContent.
lazy val unorderedSequenceContent = prod("unorderedSequenceContent") {
lazy val uoseq = self.unorderedSeq.get
UnorderedSequenceCombinator(this, uoseq.terms)
}
Should the above delay the reference to self.unorderedSeq until it's inside the UnorderedSequenceCombinator? This would delay reference to SequenceKind and any construction of the UnorderedSequence object until the 'parser' of UnorderedSequenceCombinator is referenced/used.
Actually, this appears to be an issue for the grammar as well. SequenceGrammarMixin references SequenceKind in order to determine which 'content' to use. Does this suggest then that really the UnorderedSequence should use the SequenceCombinator? The only thing that differs would be the 'body' of the object we pass to the underlying parser. And at the time that the Combinator is used, is only when the object needs to exist.
________________________________
From: Mike Beckerle
Sent: Friday, November 10, 2017 8:21:03 AM
To: dev@daffodil.apache.org; Taylor Wise
Subject: Stack overflows, UnorderedSequences, DSOM
So I've had some stack-overflows while debugging of late.
This has lead to thinking about the reasons why. There are some principles of operation for the code-base of Daffodil that are not articulated, but that need to be followed to stay out of trouble.
I am planning to add package.scala files describing these to the code base.
For DSOM one is this:
* DSOM components should be selected and constructed without reference to DFDL annotations, (including DFDL properties) carried on any schema component.
This avoids a chicken/egg situation. The DSOM graph is needed in order to implement DFDL's property scoping rules; so you can't invoke those mechanisms in order to build the DSOM graph itself, or you will very likely end up with a circularity/stack-overflow.
You can; however, look at the XML and its attributes. E.g., the XSD ref property, which is a QName for a global element/type/group. There are also some DFDL non-properties - cannot be put into scope - that are needed at the time DSOM objects are being constructed. One example is the dfdl:hiddenGroupRef "property" on xs:sequence XML elements. In general it is acceptable to look at, and even resolve global QName references at DSOM graph construction time.
Coming back to the issue of unordered sequences...
The above suggests a design flaw in DSOM currently, which is the UnorderedSequence class. In order to decide whether to construct this class or just a regular Sequence class instance, you have to examine the dfdl:sequenceKind property, which can be scoped.
Hence, I claim, the UnorderedSequence class should not exist in current form, but an isUnordered attribute on Sequence should provide the information about whether the class is ordered or not, and that is determined only *after* the DSOM graph is constructed. Once the DSOM Sequence class has been constructed, then regular (usually lazy val) class members on Sequence can construct any helper objects that are needed for dealing with the complexity of compilation for unordered sequences.
Comments?
Re: Stack overflows, UnorderedSequences, DSOM
Posted by Taylor Wise <tw...@tresys.com>.
Also, at what point is it safe to construct these helper objects? When is DSOM done?
For instance, if you go to SequenceGrammarMixin you'll see that we have a bunch of lazy evaluations concerning orderedSequenceContent and unorderedSequenceContent.
lazy val unorderedSequenceContent = prod("unorderedSequenceContent") {
lazy val uoseq = self.unorderedSeq.get
UnorderedSequenceCombinator(this, uoseq.terms)
}
Should the above delay the reference to self.unorderedSeq until it's inside the UnorderedSequenceCombinator? This would delay reference to SequenceKind and any construction of the UnorderedSequence object until the 'parser' of UnorderedSequenceCombinator is referenced/used.
Actually, this appears to be an issue for the grammar as well. SequenceGrammarMixin references SequenceKind in order to determine which 'content' to use. Does this suggest then that really the UnorderedSequence should use the SequenceCombinator? The only thing that differs would be the 'body' of the object we pass to the underlying parser. And at the time that the Combinator is used, is only when the object needs to exist.
________________________________
From: Mike Beckerle
Sent: Friday, November 10, 2017 8:21:03 AM
To: dev@daffodil.apache.org; Taylor Wise
Subject: Stack overflows, UnorderedSequences, DSOM
So I've had some stack-overflows while debugging of late.
This has lead to thinking about the reasons why. There are some principles of operation for the code-base of Daffodil that are not articulated, but that need to be followed to stay out of trouble.
I am planning to add package.scala files describing these to the code base.
For DSOM one is this:
* DSOM components should be selected and constructed without reference to DFDL annotations, (including DFDL properties) carried on any schema component.
This avoids a chicken/egg situation. The DSOM graph is needed in order to implement DFDL's property scoping rules; so you can't invoke those mechanisms in order to build the DSOM graph itself, or you will very likely end up with a circularity/stack-overflow.
You can; however, look at the XML and its attributes. E.g., the XSD ref property, which is a QName for a global element/type/group. There are also some DFDL non-properties - cannot be put into scope - that are needed at the time DSOM objects are being constructed. One example is the dfdl:hiddenGroupRef "property" on xs:sequence XML elements. In general it is acceptable to look at, and even resolve global QName references at DSOM graph construction time.
Coming back to the issue of unordered sequences...
The above suggests a design flaw in DSOM currently, which is the UnorderedSequence class. In order to decide whether to construct this class or just a regular Sequence class instance, you have to examine the dfdl:sequenceKind property, which can be scoped.
Hence, I claim, the UnorderedSequence class should not exist in current form, but an isUnordered attribute on Sequence should provide the information about whether the class is ordered or not, and that is determined only *after* the DSOM graph is constructed. Once the DSOM Sequence class has been constructed, then regular (usually lazy val) class members on Sequence can construct any helper objects that are needed for dealing with the complexity of compilation for unordered sequences.
Comments?