Why am I getting a warning message about differing alignments?

["Costello, Roger L." <co...@mitre.org>]

Hello DFDL community,

I am getting this weird warning message:

[warning] Schema Definition Warning: Continuation_Word_2 is an optional element or a variable-occurrence array and its alignment (1) is not the same as group[5]'s alignment (8).

The input seems to be getting parsed properly, and yet I get that warning message. Why am I getting that warning message? What can I do to get rid of it?

Also, the warning message refers to group[5]. Here is group[5]:

<xs:group name="hiddenParityGroup">
    <xs:sequence>
        <xs:element name="HiddenParity"
                        type="unsignedint5"
                        dfdl:outputValueCalc="{0}" />
    </xs:sequence>
</xs:group>

How does that group have any relevance?

/Roger

Re: Why am I getting a warning message about differing alignments?

["Beckerle, Mike" <mb...@tresys.com>]

I will answer your question and ask one.


This warning is about the fact that the different arms of a choice don't have the same alignment. Same thing can happen if you have a variable-occurrences element, and the group/element following it doesn't have the same alignment as the begining of the variable-occurrences element.


They may just align, by design or by luck. But the possibility is there that they won't align.


That means that as we go through the alternatives of a choice (sequentially), we actually have to, potentially, jump forward and back with respect to where we even start each branch. Similarly, for the optional case, we get to try to parse occurrences of the element, but if we find there are none, we have to then re-align to start parsing the group/element after that variable-occurrence element.


My experience of binary data, echoed by others, is that if your format specifies this, you have almost certainly made a mistake in your format description, and are not representing the format correctly.


It's not an error, because in principle, it *could* work. It's one of the things DFDL allows by way of composition properties that let you create compositions of features that may have never actually been used in any data format ever.  So we offer a warning because we think you are probably making a mistake.


The problem is that the data can fundamentally be ambiguous.


Imagine you have an array element (optional) followed by a group as in your example.


The array has alignment 1, the group alignment is 8.


When you unparse this data, suppose the array has no occurrences. So you don't put down any array occurrences into the output data stream, you align to 8 bit boundary (putting down some fill-byte bits), and then start unparsing the group.


Now imagine parsing this data. At the point where it starts attempting to parse the array element, it has alignment 1, so no bits are skipped yet, and it tries to parse an array occurrence. It's possible that the bits (which came from the fillByte from unparsing), and the start of the group, just so happen to look like the start of a valid occurrence of the array element. The parser can in fact be fooled here depending on the fillByte, and depending on what the start of an array occurrence looks like compared to the group following the array.


So modeling data when you have alignments that are not compatible for two different things that could appear at the same point in the data..... is generally problematic. But if you know your data, you may know that it's simply not possible for this ambiguity to occur. So we issue a warning, and it is "user beware".


This issue is a lot like the issue of using delimiters in binary data. in principle binary data can contain any bit pattern, so use of delimiters in binary data is always subject to the schema author knowing that the delimiter, as a bit pattern, cannot appear in the data. This is again a user-beware situation.


Warnings can be suppressed in Daffodil, btw, so if you get a warning and really do know what you are doing is NOT a mistake, the warning can be suppressed.


So my guess here in your specific case, is that this hiddenParityGroup is defined in a different file from the rest of the schema, and that file has alignment 1 and alignmentUnits 'bytes', not 'bits'. So this 5 bits of parity is going to insist on being byte-aligned, which is very weird for a field of less than 8 bits.


Now for my question:


What reasonably short warning message or addition to the message would be an improvement to make this clearer?




________________________________
From: Costello, Roger L. <co...@mitre.org>
Sent: Monday, July 29, 2019 2:25:56 PM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Why am I getting a warning message about differing alignments?


Hello DFDL community,



I am getting this weird warning message:



[warning] Schema Definition Warning: Continuation_Word_2 is an optional element or a variable-occurrence array and its alignment (1) is not the same as group[5]'s alignment (8).



The input seems to be getting parsed properly, and yet I get that warning message. Why am I getting that warning message? What can I do to get rid of it?



Also, the warning message refers to group[5]. Here is group[5]:



<xs:group name="hiddenParityGroup">
    <xs:sequence>
        <xs:element name="HiddenParity"
                        type="unsignedint5"
                        dfdl:outputValueCalc="{0}" />
    </xs:sequence>
</xs:group>



How does that group have any relevance?



/Roger