You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Adams, Joshua" <ja...@owlcyberdefense.com> on 2021/01/11 19:55:16 UTC

NewVariableInstance Suspension Issue

I'm working on wrapping up the variable direction feature and ran into a bug that is actaully unrelated to direction.  I have a test case that is pulled from the PCAP schema where on parse 4 elements are parsed, then a 5th element combines them to form an IP address with an IVC.

        <xs:sequence>
          <xs:annotation>
            <xs:appinfo source="http://www.ogf.org/dfdl/">
              <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ ex:IPsrc }" />
            </xs:appinfo>
          </xs:annotation>
          <xs:element name="byte1" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
          <xs:sequence>
            <xs:annotation>
              <xs:appinfo source="http://www.ogf.org/dfdl/">
                <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
              </xs:appinfo>
            </xs:annotation>
            <xs:element name="byte2" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
            <xs:sequence>
              <xs:annotation>
                <xs:appinfo source="http://www.ogf.org/dfdl/">
                  <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
                </xs:appinfo>
              </xs:annotation>
              <xs:element name="byte3" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
              <xs:sequence>
                <xs:annotation>
                  <xs:appinfo source="http://www.ogf.org/dfdl/">
                    <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
                  </xs:appinfo>
                </xs:annotation>
                <xs:element name="byte4" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
              </xs:sequence>
            </xs:sequence>
          </xs:sequence>
          <xs:element name="IPsrc" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="7"
            dfdl:inputValueCalc="{ fn:concat(../ex:byte1, '.', ../ex:byte2, '.', ../ex:byte3, '.', ../ex:byte4) }" />
        </xs:sequence>

The issue I'm running into is on unparse I am attempting to use newVariableInstance to reverse the process by pulling apart the 5th IP address element.  What ends up happening is the expressions in the outputValueCalc's end up suspending, as expected, until the IPsrc element is unparsed. The problem is that while we are waiting for IPsrc to be unparsed, the newVariableInstance goes out of scope, which triggers the NewVariableInstanceEndUnparser and attempts to remove the variable instance, which hasn't been added yet as we were still waiting for the suspension to end.

I guess my question is would moving the NewVariableInstanceStart/EndUnparser's into a CombinatorUnparser fix this sort of suspension issue? I'm a little unfamiliar with the CombinatorUnparser, but there were comments in the code specifically saying that these NVIStart/End unparsers should really be implemented as a combinator unparser.  I'm guessing that since the NVI stuff would be wrapped into one parser it would suspend correctly.  If it becomes one unparser I'm a little unsure how the unparser would know when it has gone out of scope and call the appropriate clean up code.

Thoughts?

Josh

Re: NewVariableInstance Suspension Issue

Posted by "Beckerle, Mike" <mb...@owlcyberdefense.com>.
Tricky!

The newVariableInstance needs to go out of scope for the ongoing unparse, but the existing variable instances need to be captured when the UStateForSuspension (I think it's callled that) is created.

Probably the latter is what isn't happening.
________________________________
From: Adams, Joshua <ja...@owlcyberdefense.com>
Sent: Monday, January 11, 2021 2:55 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: NewVariableInstance Suspension Issue

I'm working on wrapping up the variable direction feature and ran into a bug that is actaully unrelated to direction.  I have a test case that is pulled from the PCAP schema where on parse 4 elements are parsed, then a 5th element combines them to form an IP address with an IVC.

        <xs:sequence>
          <xs:annotation>
            <xs:appinfo source="http://www.ogf.org/dfdl/">
              <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ ex:IPsrc }" />
            </xs:appinfo>
          </xs:annotation>
          <xs:element name="byte1" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
          <xs:sequence>
            <xs:annotation>
              <xs:appinfo source="http://www.ogf.org/dfdl/">
                <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
              </xs:appinfo>
            </xs:annotation>
            <xs:element name="byte2" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
            <xs:sequence>
              <xs:annotation>
                <xs:appinfo source="http://www.ogf.org/dfdl/">
                  <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
                </xs:appinfo>
              </xs:annotation>
              <xs:element name="byte3" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
              <xs:sequence>
                <xs:annotation>
                  <xs:appinfo source="http://www.ogf.org/dfdl/">
                    <dfdl:newVariableInstance ref="ex:remainingAddr" defaultValue="{ fn:substring-after($ex:remainingAddr, '.') }" />
                  </xs:appinfo>
                </xs:annotation>
                <xs:element name="byte4" type="xs:unsignedByte" dfdl:outputValueCalc="{ xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
              </xs:sequence>
            </xs:sequence>
          </xs:sequence>
          <xs:element name="IPsrc" type="xs:string" dfdl:lengthKind="explicit" dfdl:length="7"
            dfdl:inputValueCalc="{ fn:concat(../ex:byte1, '.', ../ex:byte2, '.', ../ex:byte3, '.', ../ex:byte4) }" />
        </xs:sequence>

The issue I'm running into is on unparse I am attempting to use newVariableInstance to reverse the process by pulling apart the 5th IP address element.  What ends up happening is the expressions in the outputValueCalc's end up suspending, as expected, until the IPsrc element is unparsed. The problem is that while we are waiting for IPsrc to be unparsed, the newVariableInstance goes out of scope, which triggers the NewVariableInstanceEndUnparser and attempts to remove the variable instance, which hasn't been added yet as we were still waiting for the suspension to end.

I guess my question is would moving the NewVariableInstanceStart/EndUnparser's into a CombinatorUnparser fix this sort of suspension issue? I'm a little unfamiliar with the CombinatorUnparser, but there were comments in the code specifically saying that these NVIStart/End unparsers should really be implemented as a combinator unparser.  I'm guessing that since the NVI stuff would be wrapped into one parser it would suspend correctly.  If it becomes one unparser I'm a little unsure how the unparser would know when it has gone out of scope and call the appropriate clean up code.

Thoughts?

Josh