You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@daffodil.apache.org by Roger L Costello <co...@mitre.org> on 2024/06/13 11:09:51 UTC

Warning: Counterintuitive placement detected

Hi Folks,

I need my DFDL to look-ahead:

The decimal value of the first bit of the input should be wrapped in an <A> tag if the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 9, or the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 10, otherwise the first bit of the input should be wrapped in a <B> tag.

Below is my attempt at implementing this. Daffodil gives this warning message:

Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents. (id: discouragedDiscriminatorPlacement)

Here is my DFDL:

<xs:element name="test">
    <xs:complexType>
        <xs:sequence>
            <xs:choice>
                <xs:sequence> 
                    <xs:annotation>
                        <xs:appinfo source="http://www.ogf.org/dfdl/">
                            <dfdl:discriminator testKind="pattern" testPattern="(.x7x9)|(.x7xA)" />
                        </xs:appinfo>
                    </xs:annotation>
                    <xs:element name='A' type="unsignedint1"/>
                </xs:sequence>
                <xs:sequence> 
                    <xs:element name='B' type="unsignedint1"/>
                </xs:sequence>
            </xs:choice>
            <xs:element name='C' type="unsignedint4"/>
            <xs:element name='D' type="unsignedint4"/>
        </xs:sequence>
    </xs:complexType>
</xs:element>

Re: Warning: Counterintuitive placement detected

Posted by Mike Beckerle <mb...@apache.org>.
Well, here's what ChatCPT4.0 said when I asked "Explain the importance of
discriminators in DFDL"
Importance of Discriminators in DFDL

   1.

   *Differentiating Choices*:
   - *Handling Multiple Possibilities*: In many data formats, a single
      field can represent different types of information based on context. For
      example, a field could be an integer in one context and a string in
      another. Discriminators help in distinguishing between these
possibilities.
      - *Choice Elements*: DFDL supports choice elements, where a
      particular element can be one of several types. Discriminators
are used to
      determine which type is currently being parsed or serialized.
   2.

   *Ensuring Accurate Parsing*:
   - *Type Safety*: Discriminators ensure that the data is correctly
      interpreted according to its actual type. Without discriminators, the
      parser might misinterpret the data, leading to errors and incorrect
      processing.
      - *Validation*: They help in validating the data against the expected
      format, ensuring that the data conforms to the defined structure.
   3.

   *Improving Readability and Maintenance*:
   - *Explicit Conditions*: Discriminators make the conditions under which
      different data structures are used explicit. This improves the
readability
      of the DFDL schema and makes it easier to maintain and understand.
      - *Documentation*: They act as documentation within the schema,
      clearly indicating the criteria for selecting different data structures.
   4.

   *Efficiency in Processing*:
   - *Optimized Parsing*: By using discriminators, the parser can quickly
      determine the correct type of data to parse, making the parsing process
      more efficient.
      - *Reduced Ambiguity*: Discriminators reduce ambiguity in the data
      format, leading to fewer errors and less need for complex error handling
      mechanisms.

I particularly appreciated its point about readability and maintenance.
Their use of the term "validation" is not consistent with XML/DFDL usage
however.

ChatGPT4 then proceeded to hallucinate some things, including some
imaginary incorrect DFDL that doesn't actually illustrate the important
cases for discriminators.

A discriminator provides more-specific information than just an assertion.
The notion of discriminators implies something about exclusion of other
possibilities.

Toy examples aren't terribly informative. I would refer you to
https://github.com/DFDLSchemas/IBM4690-TLOG/tree/daffodil which is the TLOG
(think "cash register data") for numerous examples of discriminators in
use. Many of them are things one might have expressed using
choiceDispatchKey/choiceBranchKey, but some are not.

To me the best argument for discriminators is that most format
specifications often have prose that directly maps to the DFDL concept of
discriminators.

Consider a message format where there are a couple fields described (in the
spec) that seem to be in every message.
If the spec just says that a particular message has some field with value A
and some other field with value B, that's an assertion. It's one sided
because nothing said that no other message shares those same field values.
You'd have to study all the other message definitions to know.

But, If the specification says (or implies, perhaps by calling the fields
"type" and "sub-type") that no other message can have those same field
values, that's a discriminator. It goes beyond an assertion by giving you
not just positive value information but also exclusion information about
those fields and values.

A discriminator also has a completeness aspect. If I have two fields A and
B, which will have values 5 and 6 respectively, then the following
conceptual assertions are equivalent:

    assert(A = 5) ; assert(B = 6) // two assertions in a row are equivalent
to an "AND" of the two
    assert(A = 5 && B = 6)

If I tell you A = 5 and B = 6 form a discriminator, then the two fields
(and values) are NOT individually discriminators. Only both taken together
are a discriminator. so these are equivalent:

    assert(A = 5); discriminator(B = 6)
    assert(A = 5); assert(B = 6) ; discriminator(true);
    discriminator(A = 5 && B = 6)

Toy examples aren't very informative, but here's a DFDL schema that
produces different parse results if I use an assertion vs discriminator.
Assume dfdl:lengthKind="delimited".

<choice>
     <choice>
            <sequence dfdl:separator=";">
              <element name="tag" type="xs:string"/>
                   <annotation><appinfo source="http://www.ogf.org/dfdl/">
                      <dfdl:assert>{ . eq "A"}</dfdl:assert>
                   </appinfo></application>
              </element>
              <element name="num" type="xs:int"/>
            </sequence>
            ... more sequences here for tag B, tag C, etc....
            <sequence dfdl:separator=";">
               <element name="unrecognizedTag" type="xs:string"/>
               <element name="data" type="xs:string"/>
            </sequence>

     </choice>
     <element name="malformed" type="xs:string"/>
</choice>

If the input data is "A;5" this schema produces <tag>A</tag><num>5</num>

With the assertion as written, if the input data is "A;X" this schema
produces
<unrecognizedTag>A</unrecognizedTag><data>X</data>
This is arguably incorrect because the tag A was in fact recognized. The
failure occurred after that in the subsequent field 'num'.

If I change the dfdl:assert(s) to discriminators, this schema produces
<malformed>A;X</malformed>.


On Fri, Jun 14, 2024 at 3:41 AM Roger L Costello <co...@mitre.org> wrote:

> Mike wrote:
>
>
>
>    - Your second assert should be a discriminator.
>
>
>
> Mike, you are saying that instead of this:
>
>
>
> <xs:element name='D' type="unsignedint4">
>     <xs:annotation>
>         <xs:appinfo source=http://www.ogf.org/dfdl/>
>             <dfdl:assert>{(. eq 9) or (. eq 10)}</dfdl:assert>
>         </xs:appinfo>
>     </xs:annotation>
> </xs:element>
>
>
>
> I should use this:
>
>
>
> <xs:element name='D' type="unsignedint4">
>     <xs:annotation>
>         <xs:appinfo source=http://www.ogf.org/dfdl/>
>             <dfdl:discriminator test="{(. eq 9) or (. eq 10)}" />
>         </xs:appinfo>
>     </xs:annotation>
> </xs:element>
>
>
>
> Would you explain the difference between those two, please?
>
>
>
> Would you provide an example of where the former would behave incorrectly
> whereas the latter would behave correctly, please?
>
>
>
> /Roger
>
>
>
>
>
> *From:* Mike Beckerle <mb...@apache.org>
> *Sent:* Thursday, June 13, 2024 2:30 PM
> *To:* users@daffodil.apache.org
> *Subject:* [EXT] Re: Warning: Counterintuitive placement detected
>
>
>
> Minor suggestion. Your second assert should be a discriminator. This would
> be a more precise expression of exactly the criteria for choosing this
> sequence as the data format. DFDL can use that to improve diagnostic
> behavior if both "assertions"
>
> ZjQcmQRYFpfptBannerStart
>
> Minor suggestion.
>
>
>
> Your second assert should be a discriminator.
>
>
>
> This would be a more precise expression of exactly the criteria for
> choosing this sequence as the data format.
>
>
>
> DFDL can use that to improve diagnostic behavior if both "assertions"
> hold, but something is wrong with the data for elements B, C, and D.
>
>
>
> There is even a small possibility of an incorrect false-positive parse
> happening that the discriminator rules out.
>
>
>
>
>
> On Thu, Jun 13, 2024 at 7:52 AM Roger L Costello <co...@mitre.org>
> wrote:
>
> I found a solution that seems to work. See below. Any suggestions for
> improving it is welcome.
>
> <xs:element name="test">
>     <xs:complexType>
>         <xs:sequence>
>             <!--
>                 The decimal value of the first bit of the input should be
>                 wrapped in an <A> tag if the decimal value of the next 4
> bits
>                 equals 7 and the decimal value of the 4 bits after that
> equals 9,
>                 or the decimal value of the next 4 bits equals 7 and the
> decimal
>                 value of the 4 bits after that equals 10, otherwise the
> first bit
>                 of the input should be wrapped in a <B> tag.
>
>                 The below is not looking ahead. Assume the input is ACD.
> If C is not
>                 equal to 7 and D is not equal to 9 or 10, then we realize
> that
>                 we made an error, back up, and try the next branch of the
> choice.
>             -->
>             <xs:choice>
>                 <xs:sequence>
>                     <xs:element name='A' type="unsignedint1"/>
>                     <xs:element name='C' type="unsignedint4">
>                         <xs:annotation>
>                             <xs:appinfo source="http://www.ogf.org/dfdl/">
>                                 <dfdl:assert>{. eq 7}</dfdl:assert>
>                             </xs:appinfo>
>                         </xs:annotation>
>                     </xs:element>
>                     <xs:element name='D' type="unsignedint4">
>                         <xs:annotation>
>                             <xs:appinfo source="http://www.ogf.org/dfdl/">
>                                 <dfdl:assert>{(. eq 9) or (. eq
> 10)}</dfdl:assert>
>                             </xs:appinfo>
>                         </xs:annotation>
>                     </xs:element>
>                 </xs:sequence>
>                 <xs:sequence>
>                     <xs:element name='B' type="unsignedint1"/>
>                     <xs:element name='C' type="unsignedint4"/>
>                     <xs:element name='D' type="unsignedint4"/>
>                 </xs:sequence>
>             </xs:choice>
>         </xs:sequence>
>     </xs:complexType>
> </xs:element>
>
> -----Original Message-----
> From: Roger L Costello <co...@mitre.org>
> Sent: Thursday, June 13, 2024 7:10 AM
> To: users@daffodil.apache.org
> Subject: Warning: Counterintuitive placement detected
>
> Hi Folks,
>
> I need my DFDL to look-ahead:
>
> The decimal value of the first bit of the input should be wrapped in an
> <A> tag if the decimal value of the next 4 bits equals 7 and the decimal
> value of the 4 bits after that equals 9, or the decimal value of the next 4
> bits equals 7 and the decimal value of the 4 bits after that equals 10,
> otherwise the first bit of the input should be wrapped in a <B> tag.
>
> Below is my attempt at implementing this. Daffodil gives this warning
> message:
>
> Schema Definition Warning: Counterintuitive placement detected. Wrap the
> discriminator or assert in an empty sequence to evaluate before the
> contents. (id: discouragedDiscriminatorPlacement)
>
> Here is my DFDL:
>
> <xs:element name="test">
>     <xs:complexType>
>         <xs:sequence>
>             <xs:choice>
>                 <xs:sequence>
>                     <xs:annotation>
>                         <xs:appinfo source="http://www.ogf.org/dfdl/">
>                             <dfdl:discriminator testKind="pattern"
> testPattern="(.x7x9)|(.x7xA)" />
>                         </xs:appinfo>
>                     </xs:annotation>
>                     <xs:element name='A' type="unsignedint1"/>
>                 </xs:sequence>
>                 <xs:sequence>
>                     <xs:element name='B' type="unsignedint1"/>
>                 </xs:sequence>
>             </xs:choice>
>             <xs:element name='C' type="unsignedint4"/>
>             <xs:element name='D' type="unsignedint4"/>
>         </xs:sequence>
>     </xs:complexType>
> </xs:element>
>
>

Re: Warning: Counterintuitive placement detected

Posted by Roger L Costello <co...@mitre.org>.
Mike wrote:


  *   Your second assert should be a discriminator.

Mike, you are saying that instead of this:

<xs:element name='D' type="unsignedint4">
    <xs:annotation>
        <xs:appinfo source=http://www.ogf.org/dfdl/>
            <dfdl:assert>{(. eq 9) or (. eq 10)}</dfdl:assert>
        </xs:appinfo>
    </xs:annotation>
</xs:element>

I should use this:

<xs:element name='D' type="unsignedint4">
    <xs:annotation>
        <xs:appinfo source=http://www.ogf.org/dfdl/>
            <dfdl:discriminator test="{(. eq 9) or (. eq 10)}" />
        </xs:appinfo>
    </xs:annotation>
</xs:element>

Would you explain the difference between those two, please?

Would you provide an example of where the former would behave incorrectly whereas the latter would behave correctly, please?

/Roger


From: Mike Beckerle <mb...@apache.org>
Sent: Thursday, June 13, 2024 2:30 PM
To: users@daffodil.apache.org
Subject: [EXT] Re: Warning: Counterintuitive placement detected

Minor suggestion. Your second assert should be a discriminator. This would be a more precise expression of exactly the criteria for choosing this sequence as the data format. DFDL can use that to improve diagnostic behavior if both "assertions"
ZjQcmQRYFpfptBannerStart
Minor suggestion.

Your second assert should be a discriminator.

This would be a more precise expression of exactly the criteria for choosing this sequence as the data format.

DFDL can use that to improve diagnostic behavior if both "assertions" hold, but something is wrong with the data for elements B, C, and D.

There is even a small possibility of an incorrect false-positive parse happening that the discriminator rules out.


On Thu, Jun 13, 2024 at 7:52 AM Roger L Costello <co...@mitre.org>> wrote:
I found a solution that seems to work. See below. Any suggestions for improving it is welcome.

<xs:element name="test">
    <xs:complexType>
        <xs:sequence>
            <!--
                The decimal value of the first bit of the input should be
                wrapped in an <A> tag if the decimal value of the next 4 bits
                equals 7 and the decimal value of the 4 bits after that equals 9,
                or the decimal value of the next 4 bits equals 7 and the decimal
                value of the 4 bits after that equals 10, otherwise the first bit
                of the input should be wrapped in a <B> tag.

                The below is not looking ahead. Assume the input is ACD. If C is not
                equal to 7 and D is not equal to 9 or 10, then we realize that
                we made an error, back up, and try the next branch of the choice.
            -->
            <xs:choice>
                <xs:sequence>
                    <xs:element name='A' type="unsignedint1"/>
                    <xs:element name='C' type="unsignedint4">
                        <xs:annotation>
                            <xs:appinfo source="http://www.ogf.org/dfdl/">
                                <dfdl:assert>{. eq 7}</dfdl:assert>
                            </xs:appinfo>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name='D' type="unsignedint4">
                        <xs:annotation>
                            <xs:appinfo source="http://www.ogf.org/dfdl/">
                                <dfdl:assert>{(. eq 9) or (. eq 10)}</dfdl:assert>
                            </xs:appinfo>
                        </xs:annotation>
                    </xs:element>
                </xs:sequence>
                <xs:sequence>
                    <xs:element name='B' type="unsignedint1"/>
                    <xs:element name='C' type="unsignedint4"/>
                    <xs:element name='D' type="unsignedint4"/>
                </xs:sequence>
            </xs:choice>
        </xs:sequence>
    </xs:complexType>
</xs:element>

-----Original Message-----
From: Roger L Costello <co...@mitre.org>>
Sent: Thursday, June 13, 2024 7:10 AM
To: users@daffodil.apache.org<ma...@daffodil.apache.org>
Subject: Warning: Counterintuitive placement detected

Hi Folks,

I need my DFDL to look-ahead:

The decimal value of the first bit of the input should be wrapped in an <A> tag if the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 9, or the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 10, otherwise the first bit of the input should be wrapped in a <B> tag.

Below is my attempt at implementing this. Daffodil gives this warning message:

Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents. (id: discouragedDiscriminatorPlacement)

Here is my DFDL:

<xs:element name="test">
    <xs:complexType>
        <xs:sequence>
            <xs:choice>
                <xs:sequence>
                    <xs:annotation>
                        <xs:appinfo source="http://www.ogf.org/dfdl/">
                            <dfdl:discriminator testKind="pattern" testPattern="(.x7x9)|(.x7xA)" />
                        </xs:appinfo>
                    </xs:annotation>
                    <xs:element name='A' type="unsignedint1"/>
                </xs:sequence>
                <xs:sequence>
                    <xs:element name='B' type="unsignedint1"/>
                </xs:sequence>
            </xs:choice>
            <xs:element name='C' type="unsignedint4"/>
            <xs:element name='D' type="unsignedint4"/>
        </xs:sequence>
    </xs:complexType>
</xs:element>

Re: Warning: Counterintuitive placement detected

Posted by Mike Beckerle <mb...@apache.org>.
Minor suggestion.

Your second assert should be a discriminator.

This would be a more precise expression of exactly the criteria for
choosing this sequence as the data format.

DFDL can use that to improve diagnostic behavior if both "assertions" hold,
but something is wrong with the data for elements B, C, and D.

There is even a small possibility of an incorrect false-positive parse
happening that the discriminator rules out.


On Thu, Jun 13, 2024 at 7:52 AM Roger L Costello <co...@mitre.org> wrote:

> I found a solution that seems to work. See below. Any suggestions for
> improving it is welcome.
>
> <xs:element name="test">
>     <xs:complexType>
>         <xs:sequence>
>             <!--
>                 The decimal value of the first bit of the input should be
>                 wrapped in an <A> tag if the decimal value of the next 4
> bits
>                 equals 7 and the decimal value of the 4 bits after that
> equals 9,
>                 or the decimal value of the next 4 bits equals 7 and the
> decimal
>                 value of the 4 bits after that equals 10, otherwise the
> first bit
>                 of the input should be wrapped in a <B> tag.
>
>                 The below is not looking ahead. Assume the input is ACD.
> If C is not
>                 equal to 7 and D is not equal to 9 or 10, then we realize
> that
>                 we made an error, back up, and try the next branch of the
> choice.
>             -->
>             <xs:choice>
>                 <xs:sequence>
>                     <xs:element name='A' type="unsignedint1"/>
>                     <xs:element name='C' type="unsignedint4">
>                         <xs:annotation>
>                             <xs:appinfo source="http://www.ogf.org/dfdl/">
>                                 <dfdl:assert>{. eq 7}</dfdl:assert>
>                             </xs:appinfo>
>                         </xs:annotation>
>                     </xs:element>
>                     <xs:element name='D' type="unsignedint4">
>                         <xs:annotation>
>                             <xs:appinfo source="http://www.ogf.org/dfdl/">
>                                 <dfdl:assert>{(. eq 9) or (. eq
> 10)}</dfdl:assert>
>                             </xs:appinfo>
>                         </xs:annotation>
>                     </xs:element>
>                 </xs:sequence>
>                 <xs:sequence>
>                     <xs:element name='B' type="unsignedint1"/>
>                     <xs:element name='C' type="unsignedint4"/>
>                     <xs:element name='D' type="unsignedint4"/>
>                 </xs:sequence>
>             </xs:choice>
>         </xs:sequence>
>     </xs:complexType>
> </xs:element>
>
> -----Original Message-----
> From: Roger L Costello <co...@mitre.org>
> Sent: Thursday, June 13, 2024 7:10 AM
> To: users@daffodil.apache.org
> Subject: Warning: Counterintuitive placement detected
>
> Hi Folks,
>
> I need my DFDL to look-ahead:
>
> The decimal value of the first bit of the input should be wrapped in an
> <A> tag if the decimal value of the next 4 bits equals 7 and the decimal
> value of the 4 bits after that equals 9, or the decimal value of the next 4
> bits equals 7 and the decimal value of the 4 bits after that equals 10,
> otherwise the first bit of the input should be wrapped in a <B> tag.
>
> Below is my attempt at implementing this. Daffodil gives this warning
> message:
>
> Schema Definition Warning: Counterintuitive placement detected. Wrap the
> discriminator or assert in an empty sequence to evaluate before the
> contents. (id: discouragedDiscriminatorPlacement)
>
> Here is my DFDL:
>
> <xs:element name="test">
>     <xs:complexType>
>         <xs:sequence>
>             <xs:choice>
>                 <xs:sequence>
>                     <xs:annotation>
>                         <xs:appinfo source="http://www.ogf.org/dfdl/">
>                             <dfdl:discriminator testKind="pattern"
> testPattern="(.x7x9)|(.x7xA)" />
>                         </xs:appinfo>
>                     </xs:annotation>
>                     <xs:element name='A' type="unsignedint1"/>
>                 </xs:sequence>
>                 <xs:sequence>
>                     <xs:element name='B' type="unsignedint1"/>
>                 </xs:sequence>
>             </xs:choice>
>             <xs:element name='C' type="unsignedint4"/>
>             <xs:element name='D' type="unsignedint4"/>
>         </xs:sequence>
>     </xs:complexType>
> </xs:element>
>

RE: Warning: Counterintuitive placement detected

Posted by Roger L Costello <co...@mitre.org>.
I found a solution that seems to work. See below. Any suggestions for improving it is welcome.

<xs:element name="test">
    <xs:complexType>
        <xs:sequence>
            <!-- 
                The decimal value of the first bit of the input should be
                wrapped in an <A> tag if the decimal value of the next 4 bits
                equals 7 and the decimal value of the 4 bits after that equals 9,
                or the decimal value of the next 4 bits equals 7 and the decimal
                value of the 4 bits after that equals 10, otherwise the first bit
                of the input should be wrapped in a <B> tag.
                
                The below is not looking ahead. Assume the input is ACD. If C is not
                equal to 7 and D is not equal to 9 or 10, then we realize that
                we made an error, back up, and try the next branch of the choice.
            -->
            <xs:choice>
                <xs:sequence> 
                    <xs:element name='A' type="unsignedint1"/>
                    <xs:element name='C' type="unsignedint4">
                        <xs:annotation>
                            <xs:appinfo source="http://www.ogf.org/dfdl/">
                                <dfdl:assert>{. eq 7}</dfdl:assert>
                            </xs:appinfo>
                        </xs:annotation>
                    </xs:element>
                    <xs:element name='D' type="unsignedint4">
                        <xs:annotation>
                            <xs:appinfo source="http://www.ogf.org/dfdl/">
                                <dfdl:assert>{(. eq 9) or (. eq 10)}</dfdl:assert>
                            </xs:appinfo>
                        </xs:annotation>
                    </xs:element>
                </xs:sequence>
                <xs:sequence> 
                    <xs:element name='B' type="unsignedint1"/>
                    <xs:element name='C' type="unsignedint4"/>
                    <xs:element name='D' type="unsignedint4"/>
                </xs:sequence>
            </xs:choice>
        </xs:sequence>
    </xs:complexType>
</xs:element>

-----Original Message-----
From: Roger L Costello <co...@mitre.org> 
Sent: Thursday, June 13, 2024 7:10 AM
To: users@daffodil.apache.org
Subject: Warning: Counterintuitive placement detected

Hi Folks,

I need my DFDL to look-ahead:

The decimal value of the first bit of the input should be wrapped in an <A> tag if the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 9, or the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 10, otherwise the first bit of the input should be wrapped in a <B> tag.

Below is my attempt at implementing this. Daffodil gives this warning message:

Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents. (id: discouragedDiscriminatorPlacement)

Here is my DFDL:

<xs:element name="test">
    <xs:complexType>
        <xs:sequence>
            <xs:choice>
                <xs:sequence> 
                    <xs:annotation>
                        <xs:appinfo source="http://www.ogf.org/dfdl/">
                            <dfdl:discriminator testKind="pattern" testPattern="(.x7x9)|(.x7xA)" />
                        </xs:appinfo>
                    </xs:annotation>
                    <xs:element name='A' type="unsignedint1"/>
                </xs:sequence>
                <xs:sequence> 
                    <xs:element name='B' type="unsignedint1"/>
                </xs:sequence>
            </xs:choice>
            <xs:element name='C' type="unsignedint4"/>
            <xs:element name='D' type="unsignedint4"/>
        </xs:sequence>
    </xs:complexType>
</xs:element>

Re: Warning: Counterintuitive placement detected

Posted by Mike Beckerle <mb...@apache.org>.
So our warning is not correct here. It doesn't distinguish
discriminators/asserts with test kind 'pattern' from those with text kind
'expression'.

The DFDL spec Section 9.5 Evaluation Order for Statement Annotations says
that statement annotations on a sequence execute

1. dfdl:discriminator or dfdl:assert(s) with testKind 'pattern' (parsing
only)
2. dfdl:newVariableInstance(s) - in lexical order, innermost schema
component first
3. dfdl:setVariable(s) - in lexical order, innermost schema component first
4. dfdl:sequence or dfdl:choice or dfdl:group following property scoping
rules and evaluating any property expressions (corresponds to
ComplexContent grammar region)
5. dfdl:discriminator or dfdl:assert(s) with testKind 'expression' (parsing
only)

Step 4 is the parsing of the contents of the sequence.

1, 2, and 3 happen before the sequence contents are parsed. Only 5 happens
after. So only 5, written at the top of a sequence with some content
besides just the annotation, has the counterintuitive placement issue where
the sequence body happens first, then the discriminator/assert that appears
lexically before it.

There was some rationale for this odd rule that I can't crisply recall,
having to do with schema authors being able to write discriminators without
having to worry about whether they are forward referencing or not.

But a pattern discriminator should not be triggering the warning. I'll
submit a bug report on this.



On Thu, Jun 13, 2024 at 8:12 AM Steve Lawrence <sl...@apache.org> wrote:

> Discriminators/asserts annotated on sequences are evaluated at the end of
> the
> sequence after the content has been evaluated. This is usually not what is
> intended so Daffodil outputs this warning. To get it to evaluate before
> the
> content (like what you intend in this case) you need to wrap the
> discriminator/assert in a sequence, for example:
>
>    <xs:sequence>
>      <xs:sequence>
>        <xs:annotation>
>          <xs:appinfo source="http://www.ogf.org/dfdl/">
>            <dfdl:discriminator ... />
>          </xs:appinfo>
>        </xs:annotation>
>      </xs:sequence>
>      <xs:element name='A' type="unsignedint1"/>
>   </xs:sequence>
>
> Tha should resolve the warning.
>
> Also, note that lengthKind="pattern" is character based, so might not work
> as
> expected since your data is bit-based. In order for a pattern
> discriminator to
> work you need to decode 4-bit characters (since both fields you care about
> are 4
> bits) after skipping the first bit. We have the X-DFDL-HEX-MSBF encoding
> that
> can decode 4-bit characters to a hex string, but there is no way to skip
> the
> first bit. You could move the discriminator sequence to after the A
> element,
> which works since at that point A is already parsed and the first bit is
> skipped, e.g.
>
>    <xs:sequence>
>      <xs:element name='A' type="unsignedint1"/>
>      <xs:sequence dfdl:encoding="X-DFDL-HEX-MSBF">
>        <xs:annotation>
>          <xs:appinfo source="http://www.ogf.org/dfdl/">
>            <dfdl:discriminator testKind="pattern" testPattern="(79)|(7A)"
> />
>          </xs:appinfo>
>        </xs:annotation>
>      </xs:sequence>
>    </xs:sequence>
>
> But that feels a bit complex and unclear to me.
>
> I would suggest instead using the dfdlx:lookAhead extension function which
> was
> added exactly for this kind of thing. For example, your discriminator
> would look
> something like this:
>
>    <dfdl:discriminator testKind="expression" test="{
>      ((dfdlx:lookAhead(1, 4) eq 7) and (dfdlx:lookAhead(5, 4) eq 9)) or
>      ((dfdlx:lookAhead(1, 4) eq 7) and (dfdlx:lookAhead(5, 4) eq 10))
>    }" />
>
> It's a bit longer, but I think the intention is much ore clear. It's also
> probably going to be significantly more efficient since it doesn't have to
> decode characters and then run a regex on them.
>
>
> On 2024-06-13 07:09 AM, Roger L Costello wrote:
> > Hi Folks,
> >
> > I need my DFDL to look-ahead:
> >
> > The decimal value of the first bit of the input should be wrapped in an
> <A> tag if the decimal value of the next 4 bits equals 7 and the decimal
> value of the 4 bits after that equals 9, or the decimal value of the next 4
> bits equals 7 and the decimal value of the 4 bits after that equals 10,
> otherwise the first bit of the input should be wrapped in a <B> tag.
> >
> > Below is my attempt at implementing this. Daffodil gives this warning
> message:
> >
> > Schema Definition Warning: Counterintuitive placement detected. Wrap the
> discriminator or assert in an empty sequence to evaluate before the
> contents. (id: discouragedDiscriminatorPlacement)
> >
> > Here is my DFDL:
> >
> > <xs:element name="test">
> >      <xs:complexType>
> >          <xs:sequence>
> >              <xs:choice>
> >                  <xs:sequence>
> >                      <xs:annotation>
> >                          <xs:appinfo source="http://www.ogf.org/dfdl/">
> >                              <dfdl:discriminator testKind="pattern"
> testPattern="(.x7x9)|(.x7xA)" />
> >                          </xs:appinfo>
> >                      </xs:annotation>
> >                      <xs:element name='A' type="unsignedint1"/>
> >                  </xs:sequence>
> >                  <xs:sequence>
> >                      <xs:element name='B' type="unsignedint1"/>
> >                  </xs:sequence>
> >              </xs:choice>
> >              <xs:element name='C' type="unsignedint4"/>
> >              <xs:element name='D' type="unsignedint4"/>
> >          </xs:sequence>
> >      </xs:complexType>
> > </xs:element>
>
>

Re: Warning: Counterintuitive placement detected

Posted by Steve Lawrence <sl...@apache.org>.
Discriminators/asserts annotated on sequences are evaluated at the end of the 
sequence after the content has been evaluated. This is usually not what is 
intended so Daffodil outputs this warning. To get it to evaluate before the 
content (like what you intend in this case) you need to wrap the 
discriminator/assert in a sequence, for example:

   <xs:sequence>
     <xs:sequence>
       <xs:annotation>
         <xs:appinfo source="http://www.ogf.org/dfdl/">
           <dfdl:discriminator ... />
         </xs:appinfo>
       </xs:annotation>
     </xs:sequence>
     <xs:element name='A' type="unsignedint1"/>
  </xs:sequence>

Tha should resolve the warning.

Also, note that lengthKind="pattern" is character based, so might not work as 
expected since your data is bit-based. In order for a pattern discriminator to 
work you need to decode 4-bit characters (since both fields you care about are 4 
bits) after skipping the first bit. We have the X-DFDL-HEX-MSBF encoding that 
can decode 4-bit characters to a hex string, but there is no way to skip the 
first bit. You could move the discriminator sequence to after the A element, 
which works since at that point A is already parsed and the first bit is 
skipped, e.g.

   <xs:sequence>
     <xs:element name='A' type="unsignedint1"/>
     <xs:sequence dfdl:encoding="X-DFDL-HEX-MSBF">
       <xs:annotation>
         <xs:appinfo source="http://www.ogf.org/dfdl/">
           <dfdl:discriminator testKind="pattern" testPattern="(79)|(7A)" />
         </xs:appinfo>
       </xs:annotation>
     </xs:sequence>
   </xs:sequence>

But that feels a bit complex and unclear to me.

I would suggest instead using the dfdlx:lookAhead extension function which was 
added exactly for this kind of thing. For example, your discriminator would look 
something like this:

   <dfdl:discriminator testKind="expression" test="{
     ((dfdlx:lookAhead(1, 4) eq 7) and (dfdlx:lookAhead(5, 4) eq 9)) or
     ((dfdlx:lookAhead(1, 4) eq 7) and (dfdlx:lookAhead(5, 4) eq 10))
   }" />

It's a bit longer, but I think the intention is much ore clear. It's also 
probably going to be significantly more efficient since it doesn't have to 
decode characters and then run a regex on them.


On 2024-06-13 07:09 AM, Roger L Costello wrote:
> Hi Folks,
> 
> I need my DFDL to look-ahead:
> 
> The decimal value of the first bit of the input should be wrapped in an <A> tag if the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 9, or the decimal value of the next 4 bits equals 7 and the decimal value of the 4 bits after that equals 10, otherwise the first bit of the input should be wrapped in a <B> tag.
> 
> Below is my attempt at implementing this. Daffodil gives this warning message:
> 
> Schema Definition Warning: Counterintuitive placement detected. Wrap the discriminator or assert in an empty sequence to evaluate before the contents. (id: discouragedDiscriminatorPlacement)
> 
> Here is my DFDL:
> 
> <xs:element name="test">
>      <xs:complexType>
>          <xs:sequence>
>              <xs:choice>
>                  <xs:sequence>
>                      <xs:annotation>
>                          <xs:appinfo source="http://www.ogf.org/dfdl/">
>                              <dfdl:discriminator testKind="pattern" testPattern="(.x7x9)|(.x7xA)" />
>                          </xs:appinfo>
>                      </xs:annotation>
>                      <xs:element name='A' type="unsignedint1"/>
>                  </xs:sequence>
>                  <xs:sequence>
>                      <xs:element name='B' type="unsignedint1"/>
>                  </xs:sequence>
>              </xs:choice>
>              <xs:element name='C' type="unsignedint4"/>
>              <xs:element name='D' type="unsignedint4"/>
>          </xs:sequence>
>      </xs:complexType>
> </xs:element>