[jira] [Commented] (DAFFODIL-2626) Circular deadlock when computing stored length around prefixed-length elements

["Steve Lawrence (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DAFFODIL-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485985#comment-17485985 ] 

Steve Lawrence commented on DAFFODIL-2626:
------------------------------------------

I've found a fix for test_computedLengthAroundPrefixedLengths1u--we just need to implement NYI logic for lengthKind="prefixed" in the alignment algorithm. However, I'm not sure if we can optimize out the alignment in test_prefixedAroundDelimitedString1 that causes a deadlock.

Keep in mind that our alignment optimization algorithm currently only looks at lexical terms. This means that for global decls we assume we know nothing about their alignment because we do not know where they are referenced from.

Side note, I think we actually have a bug where instead of assuming no information, we assume the global decl is known to be aligned to that of whatever its model group requires. This is clearly wrong, since how and where the global decl references affects its alignment. Things just happen to work out in most cases because mixed alignment of things is not that common. I'll work on seeing if I can come up with a test case that shows this.

Back to the issue, here's a simplified example showing our inability to optimize out some alignment:
{code:xml}
  <xs:annotation>
    <xs:appinfo source="http://www.ogf.org/dfdl/">
      <dfdl:format
        ref="ex:GeneralFormat"
        representation="binary"
        alignment="1"
        alignmentUnits="bits"
      />
    </xs:appinfo>
  </xs:annotation>

  <xs:element name="root">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="field" type="xs:int"
          dfdl:lengthKind="explicit" dfdl:lengthUnits="bits" dfdl:length="8"
          dfdl:alignmentUnits="bytes" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>
{code}
In this snippet, "field" is byte aligned and everything else is bit aligned. If our alignment algorithm knew that the root element was in fact the root, then it could figure out that "field" actually starts at bitPos 0 (nothign else consumes data), and so it is already byte aligned, and alignment is not needed.

But we don't have that information. Lexically, we have no idea where "root" will be referenced from or what its alignment will be when referenced, and so we must assume no alignment. Because the sequence is 1-bit aligned, it's fine and requires no alignment. But "field" requires byte alignment. We don't have any information about the alignment here, and so we must insert an alignment parser/unparse.

This is essentially what's going on with test_prefixedAroundDelimitedString1, but ends up deadlock on unparse because of that alignment we can't optimize out

So, I'm not sure how we can fix this. We could use enclosingTerms to look at all potential alignments and combine them. And in fact this does fix the issue for this test, but it leads to a StackOverflow with many other tests. I haven't dug into this, but I assume we're getting stuck in a loop, and is likely the reason we switched to lexical-only logic.

Thoughts?

> Circular deadlock when computing stored length around prefixed-length elements
> ------------------------------------------------------------------------------
>
>                 Key: DAFFODIL-2626
>                 URL: https://issues.apache.org/jira/browse/DAFFODIL-2626
>             Project: Daffodil
>          Issue Type: Bug
>          Components: Back End
>    Affects Versions: 3.2.1
>            Reporter: Mike Beckerle
>            Assignee: Steve Lawrence
>            Priority: Critical
>
> I have a small schema with a messageLength element as the first element, which has outputValueCalc on the message payload.
> The message payload contains prefixed-length strings.
> I get a deadlock on unparsing:
> {code}
> org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: daffodil) SuspensionDeadlockException: Runtime Schema Definition Error: Expressions/Unparsers are circularly deadlocked (mutually defined):
>  - SimpleTypeRetryUnparserSuspendableOperation for messageLength
>  - ElementUnusedUnparserSuspendableOperation for payload
>  - AlignmentFillUnparserSuspendableOperation for address
> Schema context: messageLength Location line 59 column 18 in file:/tmp/s__3044777217024912267.dfdl.xsd
> Data location was preceding byte 0
> {code}
> See test test_computedLengthAroundPrefixedLengths1u
> Test test_computedLengthAroundPrefixedLengths1p is the same thing, just testing the parse direction.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)