You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@daffodil.apache.org by Patrick Grandjean <p....@gmail.com> on 2020/02/03 19:23:20 UTC

Optional element with initiator

Hi!

I'm trying to create my first DFDL schema to parse a fixed column format.
The format may change from one line to the other depending on the first
character of the line. Some lines are optional, therefore I use
minOccurs="0" and dlfl:occursCountKind="implicit". Here is the schema:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
           xmlns:ex="http://example.com"
           xmlns:ama="http://www.amadeus.com"
           targetNamespace="http://www.amadeus.com"
           elementFormDefault="unqualified">

    <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
    <xs:include schemaLocation="IT01.dfdl.xsd" />
    <xs:include schemaLocation="IT02.dfdl.xsd" />
    <xs:include schemaLocation="IT03.dfdl.xsd" />
    <xs:include schemaLocation="IT05.dfdl.xsd" />

    <xs:annotation>
        <xs:appinfo source="http://www.ogf.org/dfdl/">
            <dfdl:format alignment="1"
                         alignmentUnits="bytes"
                         encoding="ASCII"
                         encodingErrorPolicy="replace"
                         escapeSchemeRef=""
                         floating="no"
                         ignoreCase="no"
                         initiatedContent="no"
                         initiator=""
                         leadingSkip="0"
                         lengthKind="implicit"
                         lengthUnits="bytes"
                         outputNewLine="%LF;"
                         representation="text"
                         separator=""
                         separatorSuppressionPolicy="never"
                         sequenceKind="ordered"
                         terminator=""
                         textBidi="no"
                         textPadKind="none"
                         textStringJustification="center"
                         textStringPadCharacter="%WSP;"
                         textTrimKind="padChar"
                         trailingSkip="0"
                         truncateSpecifiedLengthString="no"/>
        </xs:appinfo>
    </xs:annotation>

    <xs:element name="records" type="ama:Records" />

    <xs:complexType name="Records">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it01" minOccurs="0" maxOccurs="1"
type="ama:IT01" dfdl:initiator="1" dfdl:occursCountKind="implicit" />
            <xs:element name="record" type="ama:Record" minOccurs="1"
dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="Record">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it02" type="ama:IT02" dfdl:initiator="2"
dfdl:occursCountKind="implicit" />
            <xs:element name="it03" minOccurs="0" type="ama:IT03"
dfdl:initiator="3" dfdl:occursCountKind="implicit" />
            <xs:element name="it05" minOccurs="0" type="ama:IT05"
dfdl:initiator="5" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

</xs:schema>


A file would look like this:

1......
2......
5......

Elements it01 (line starting with '1') and it02 (line starting with '2')
are parsed successfully. Element it03 is missing, but since it is optional
it is OK. The problem is that the Daffodil raises an error saying it has
not found initiator '3'. How to make the initiator optional?

Patrick.

Re: Optional element with initiator

Posted by "Beckerle, Mike" <mb...@tresys.com>.
I found the issue.

I noticed that you import the DFDLGeneralFormat.dfdl.xml, but then you don't use it.

So I checked, and in your list of property definitions in dfdl:format they are almost all the same as if you had ref="ama:GeneralFormat" except for one crucial one.

You have separatorSuppressionPolicy="never".

That's not what you want. You want "anyEmpty".

This separatorSuppressionPolicy property is among the most difficult to deal with in the spec.

The "never" value means "never suppress separators, even if the item is missing". That is really about fixed length arrays and fixed-size tuples where the separators are always going to appear even if the element content is missing.

Think sparse data formats with data like "/1,2,,,5,,7,8/" where the missing pieces are still delimited by separators.

I think the diagnostic message is not good here, and will create a bug ticket about it. It should say specifically that it was looking for the separator, not that the element initiator isn't present. Or it should say it didn't find the element because it didn't find the initiator, AND also didn't find the associated separator.

If I remove all your format properties and just use
<dfdl:format ref="ama:GeneralFormat"/> then it works.

One other minor thing: You had textStringPadCharacter="%WSP;" which is a schema-definition error. I changed it to %SP; which is probably what you intended, and that's the value in ama:GeneralFormat, so I removed it entirely.

The TDML file that has the test that shows it working is attached.


________________________________
From: Beckerle, Mike <mb...@tresys.com>
Sent: Wednesday, February 5, 2020 9:07 AM
To: Patrick Grandjean <p....@gmail.com>; users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Optional element with initiator

Sorry for duplicate. This time to users list also.

I will try to reproduce the issue and get back to you.
________________________________
From: Patrick Grandjean <p....@gmail.com>
Sent: Tuesday, February 4, 2020 2:31 PM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Optional element with initiator

Thank for the answer.

Unfortunately, if I try to remove any dfdl:occursCountKind I get an error from daffodil saying this attribute is missing.

I have removed the postfix attribute, thank you for the tip!

I have set all elements to be optional (minOccurs="0"), but I still get an Initiator exception:

diff:
  No differences
----------------------------------------------------------------- 91
parser: <Initiator/>
bitPosition: 4096
data:
            │                                        │
  87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef
  00000200: 3530 3030 3030 3130 3030 3030 3030 3230  5000001000000020
  00000210: 3030 4555 5232 2020 2020 2020 2020 3030  00EUR2␣␣␣␣␣␣␣␣00
infoset:
  <record>
    <it02>
     [....]
    </it02>
    <it03 />
  </record>
diff:
  No differences
failure:
  Parse Error: Initiator '3' not found
  Schema context: it03 Location line 71 column 14 in file:/path/to/apache-daffodil-2.5.0-incubating-bin/bin/../../src/main/resources/grammar.dfdl.xsd
  Data location was preceding byte 512

I would expect that if an initiator is not found, the parser would backtrack and try the next element in the sequence. Is there a policy to be set?

I have tried to replace the initiators with elements having fixed values, but it seems that the fixed value is not checked. Example below, with fixed="1":

<xs:element name="recordIdentifier" type="xs:string" fixed="1" dfdl:length="1" dfdl:lengthKind="explicit" />



On Mon, Feb 3, 2020 at 3:17 PM Beckerle, Mike <mb...@tresys.com>> wrote:
I see a couple issues here.

First your record element is not optional but you have dfdl:occursCountKind="implicit" on it, suggesting you wanted minOccurs="0" on it, or maxOccurs="unbounded",  but you didn't put that.

Same for it02 element, which probably wants minOccurs="0".

Second, your records type has a sequence with postfix newline separator.
Your inner sequence inside the record type *also* has postfix newline separator.

This means you will have TWO newlines required at the end of your data.

If you don't have two newlines, then it's going to backtrack and, while I can't construct a specific reason why it ends up giving you an error on it03, I can imagine that could happen.

I suggest removing the postfix from the inner sequence so it is only infix separated, and the final newline will come from the outer sequence.

Then I suggest making it02 element optional and record element optional or removing the dfdl:occursCountKind="implicit" from either or both of them.

Then try again. Post your results here again.

You may want to turn on the trace feature. You get TONS of sometimes obscure output from trace (fixing that is on our roadmap), but often one can wade through it and spot where things go wrong.

If your format has many many of these different record sub-types you will eventually want to switch away from using initiators to a choice using the dfdl:choiceDispatchKey property.

But we should get what you started with working properly first.




________________________________
From: Patrick Grandjean <p....@gmail.com>>
Sent: Monday, February 3, 2020 2:23 PM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: Optional element with initiator

Hi!

I'm trying to create my first DFDL schema to parse a fixed column format. The format may change from one line to the other depending on the first character of the line. Some lines are optional, therefore I use minOccurs="0" and dlfl:occursCountKind="implicit". Here is the schema:


<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
           xmlns:ex="http://example.com"
           xmlns:ama="http://www.amadeus.com"
           targetNamespace="http://www.amadeus.com"
           elementFormDefault="unqualified">

    <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
    <xs:include schemaLocation="IT01.dfdl.xsd" />
    <xs:include schemaLocation="IT02.dfdl.xsd" />
    <xs:include schemaLocation="IT03.dfdl.xsd" />
    <xs:include schemaLocation="IT05.dfdl.xsd" />

    <xs:annotation>
        <xs:appinfo source="http://www.ogf.org/dfdl/">
            <dfdl:format alignment="1"
                         alignmentUnits="bytes"
                         encoding="ASCII"
                         encodingErrorPolicy="replace"
                         escapeSchemeRef=""
                         floating="no"
                         ignoreCase="no"
                         initiatedContent="no"
                         initiator=""
                         leadingSkip="0"
                         lengthKind="implicit"
                         lengthUnits="bytes"
                         outputNewLine="%LF;"
                         representation="text"
                         separator=""
                         separatorSuppressionPolicy="never"
                         sequenceKind="ordered"
                         terminator=""
                         textBidi="no"
                         textPadKind="none"
                         textStringJustification="center"
                         textStringPadCharacter="%WSP;"
                         textTrimKind="padChar"
                         trailingSkip="0"
                         truncateSpecifiedLengthString="no"/>
        </xs:appinfo>
    </xs:annotation>

    <xs:element name="records" type="ama:Records" />

    <xs:complexType name="Records">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it01" minOccurs="0" maxOccurs="1" type="ama:IT01" dfdl:initiator="1" dfdl:occursCountKind="implicit" />
            <xs:element name="record" type="ama:Record" minOccurs="1" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="Record">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it02" type="ama:IT02" dfdl:initiator="2" dfdl:occursCountKind="implicit" />
            <xs:element name="it03" minOccurs="0" type="ama:IT03" dfdl:initiator="3" dfdl:occursCountKind="implicit" />
            <xs:element name="it05" minOccurs="0" type="ama:IT05" dfdl:initiator="5" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

</xs:schema>

A file would look like this:

1......
2......
5......

Elements it01 (line starting with '1') and it02 (line starting with '2') are parsed successfully. Element it03 is missing, but since it is optional it is OK. The problem is that the Daffodil raises an error saying it has not found initiator '3'. How to make the initiator optional?

Patrick.

Re: Optional element with initiator

Posted by "Beckerle, Mike" <mb...@tresys.com>.
Sorry for duplicate. This time to users list also.

I will try to reproduce the issue and get back to you.
________________________________
From: Patrick Grandjean <p....@gmail.com>
Sent: Tuesday, February 4, 2020 2:31 PM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Optional element with initiator

Thank for the answer.

Unfortunately, if I try to remove any dfdl:occursCountKind I get an error from daffodil saying this attribute is missing.

I have removed the postfix attribute, thank you for the tip!

I have set all elements to be optional (minOccurs="0"), but I still get an Initiator exception:

diff:
  No differences
----------------------------------------------------------------- 91
parser: <Initiator/>
bitPosition: 4096
data:
            │                                        │
  87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef
  00000200: 3530 3030 3030 3130 3030 3030 3030 3230  5000001000000020
  00000210: 3030 4555 5232 2020 2020 2020 2020 3030  00EUR2␣␣␣␣␣␣␣␣00
infoset:
  <record>
    <it02>
     [....]
    </it02>
    <it03 />
  </record>
diff:
  No differences
failure:
  Parse Error: Initiator '3' not found
  Schema context: it03 Location line 71 column 14 in file:/path/to/apache-daffodil-2.5.0-incubating-bin/bin/../../src/main/resources/grammar.dfdl.xsd
  Data location was preceding byte 512

I would expect that if an initiator is not found, the parser would backtrack and try the next element in the sequence. Is there a policy to be set?

I have tried to replace the initiators with elements having fixed values, but it seems that the fixed value is not checked. Example below, with fixed="1":

<xs:element name="recordIdentifier" type="xs:string" fixed="1" dfdl:length="1" dfdl:lengthKind="explicit" />



On Mon, Feb 3, 2020 at 3:17 PM Beckerle, Mike <mb...@tresys.com>> wrote:
I see a couple issues here.

First your record element is not optional but you have dfdl:occursCountKind="implicit" on it, suggesting you wanted minOccurs="0" on it, or maxOccurs="unbounded",  but you didn't put that.

Same for it02 element, which probably wants minOccurs="0".

Second, your records type has a sequence with postfix newline separator.
Your inner sequence inside the record type *also* has postfix newline separator.

This means you will have TWO newlines required at the end of your data.

If you don't have two newlines, then it's going to backtrack and, while I can't construct a specific reason why it ends up giving you an error on it03, I can imagine that could happen.

I suggest removing the postfix from the inner sequence so it is only infix separated, and the final newline will come from the outer sequence.

Then I suggest making it02 element optional and record element optional or removing the dfdl:occursCountKind="implicit" from either or both of them.

Then try again. Post your results here again.

You may want to turn on the trace feature. You get TONS of sometimes obscure output from trace (fixing that is on our roadmap), but often one can wade through it and spot where things go wrong.

If your format has many many of these different record sub-types you will eventually want to switch away from using initiators to a choice using the dfdl:choiceDispatchKey property.

But we should get what you started with working properly first.




________________________________
From: Patrick Grandjean <p....@gmail.com>>
Sent: Monday, February 3, 2020 2:23 PM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: Optional element with initiator

Hi!

I'm trying to create my first DFDL schema to parse a fixed column format. The format may change from one line to the other depending on the first character of the line. Some lines are optional, therefore I use minOccurs="0" and dlfl:occursCountKind="implicit". Here is the schema:


<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
           xmlns:ex="http://example.com"
           xmlns:ama="http://www.amadeus.com"
           targetNamespace="http://www.amadeus.com"
           elementFormDefault="unqualified">

    <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
    <xs:include schemaLocation="IT01.dfdl.xsd" />
    <xs:include schemaLocation="IT02.dfdl.xsd" />
    <xs:include schemaLocation="IT03.dfdl.xsd" />
    <xs:include schemaLocation="IT05.dfdl.xsd" />

    <xs:annotation>
        <xs:appinfo source="http://www.ogf.org/dfdl/">
            <dfdl:format alignment="1"
                         alignmentUnits="bytes"
                         encoding="ASCII"
                         encodingErrorPolicy="replace"
                         escapeSchemeRef=""
                         floating="no"
                         ignoreCase="no"
                         initiatedContent="no"
                         initiator=""
                         leadingSkip="0"
                         lengthKind="implicit"
                         lengthUnits="bytes"
                         outputNewLine="%LF;"
                         representation="text"
                         separator=""
                         separatorSuppressionPolicy="never"
                         sequenceKind="ordered"
                         terminator=""
                         textBidi="no"
                         textPadKind="none"
                         textStringJustification="center"
                         textStringPadCharacter="%WSP;"
                         textTrimKind="padChar"
                         trailingSkip="0"
                         truncateSpecifiedLengthString="no"/>
        </xs:appinfo>
    </xs:annotation>

    <xs:element name="records" type="ama:Records" />

    <xs:complexType name="Records">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it01" minOccurs="0" maxOccurs="1" type="ama:IT01" dfdl:initiator="1" dfdl:occursCountKind="implicit" />
            <xs:element name="record" type="ama:Record" minOccurs="1" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="Record">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it02" type="ama:IT02" dfdl:initiator="2" dfdl:occursCountKind="implicit" />
            <xs:element name="it03" minOccurs="0" type="ama:IT03" dfdl:initiator="3" dfdl:occursCountKind="implicit" />
            <xs:element name="it05" minOccurs="0" type="ama:IT05" dfdl:initiator="5" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

</xs:schema>

A file would look like this:

1......
2......
5......

Elements it01 (line starting with '1') and it02 (line starting with '2') are parsed successfully. Element it03 is missing, but since it is optional it is OK. The problem is that the Daffodil raises an error saying it has not found initiator '3'. How to make the initiator optional?

Patrick.

Re: Optional element with initiator

Posted by Patrick Grandjean <p....@gmail.com>.
Thank for the answer.

Unfortunately, if I try to remove any dfdl:occursCountKind I get an error
from daffodil saying this attribute is missing.

I have removed the postfix attribute, thank you for the tip!

I have set all elements to be optional (minOccurs="0"), but I still get an
Initiator exception:

diff:
  No differences
----------------------------------------------------------------- 91
parser: <Initiator/>
bitPosition: 4096
data:
            │                                        │
  87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef
  00000200: 3530 3030 3030 3130 3030 3030 3030 3230  5000001000000020
  00000210: 3030 4555 5232 2020 2020 2020 2020 3030  00EUR2␣␣␣␣␣␣␣␣00
infoset:
  <record>
    <it02>
     [....]
    </it02>
    <it03 />
  </record>
diff:
  No differences
failure:
  Parse Error: Initiator '3' not found
  Schema context: it03 Location line 71 column 14 in
file:/path/to/apache-daffodil-2.5.0-incubating-bin/bin/../../src/main/resources/grammar.dfdl.xsd
  Data location was preceding byte 512

I would expect that if an initiator is not found, the parser would
backtrack and try the next element in the sequence. Is there a policy to be
set?

I have tried to replace the initiators with elements having fixed values,
but it seems that the fixed value is not checked. Example below, with
fixed="1":

<xs:element name="recordIdentifier" type="xs:string" fixed="1" dfdl:length="1"
dfdl:lengthKind="explicit" />



On Mon, Feb 3, 2020 at 3:17 PM Beckerle, Mike <mb...@tresys.com> wrote:

> I see a couple issues here.
>
> First your record element is not optional but you have
> dfdl:occursCountKind="implicit" on it, suggesting you wanted minOccurs="0"
> on it, or maxOccurs="unbounded",  but you didn't put that.
>
> Same for it02 element, which probably wants minOccurs="0".
>
> Second, your records type has a sequence with postfix newline separator.
> Your inner sequence inside the record type *also* has postfix newline
> separator.
>
> This means you will have TWO newlines required at the end of your data.
>
> If you don't have two newlines, then it's going to backtrack and, while I
> can't construct a specific reason why it ends up giving you an error on
> it03, I can imagine that could happen.
>
> I suggest removing the postfix from the inner sequence so it is only infix
> separated, and the final newline will come from the outer sequence.
>
> Then I suggest making it02 element optional and record element optional or
> removing the dfdl:occursCountKind="implicit" from either or both of them.
>
> Then try again. Post your results here again.
>
> You may want to turn on the trace feature. You get TONS of sometimes
> obscure output from trace (fixing that is on our roadmap), but often one
> can wade through it and spot where things go wrong.
>
> If your format has many many of these different record sub-types you will
> eventually want to switch away from using initiators to a choice using the
> dfdl:choiceDispatchKey property.
>
> But we should get what you started with working properly first.
>
>
>
>
> ------------------------------
> *From:* Patrick Grandjean <p....@gmail.com>
> *Sent:* Monday, February 3, 2020 2:23 PM
> *To:* users@daffodil.apache.org <us...@daffodil.apache.org>
> *Subject:* Optional element with initiator
>
> Hi!
>
> I'm trying to create my first DFDL schema to parse a fixed column format.
> The format may change from one line to the other depending on the first
> character of the line. Some lines are optional, therefore I use
> minOccurs="0" and dlfl:occursCountKind="implicit". Here is the schema:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
>            xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
>            xmlns:ex="http://example.com"
>            xmlns:ama="http://www.amadeus.com"
>            targetNamespace="http://www.amadeus.com"
>            elementFormDefault="unqualified">
>
>     <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
>     <xs:include schemaLocation="IT01.dfdl.xsd" />
>     <xs:include schemaLocation="IT02.dfdl.xsd" />
>     <xs:include schemaLocation="IT03.dfdl.xsd" />
>     <xs:include schemaLocation="IT05.dfdl.xsd" />
>
>     <xs:annotation>
>         <xs:appinfo source="http://www.ogf.org/dfdl/">
>             <dfdl:format alignment="1"
>                          alignmentUnits="bytes"
>                          encoding="ASCII"
>                          encodingErrorPolicy="replace"
>                          escapeSchemeRef=""
>                          floating="no"
>                          ignoreCase="no"
>                          initiatedContent="no"
>                          initiator=""
>                          leadingSkip="0"
>                          lengthKind="implicit"
>                          lengthUnits="bytes"
>                          outputNewLine="%LF;"
>                          representation="text"
>                          separator=""
>                          separatorSuppressionPolicy="never"
>                          sequenceKind="ordered"
>                          terminator=""
>                          textBidi="no"
>                          textPadKind="none"
>                          textStringJustification="center"
>                          textStringPadCharacter="%WSP;"
>                          textTrimKind="padChar"
>                          trailingSkip="0"
>                          truncateSpecifiedLengthString="no"/>
>         </xs:appinfo>
>     </xs:annotation>
>
>     <xs:element name="records" type="ama:Records" />
>
>     <xs:complexType name="Records">
>         <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
>             <xs:element name="it01" minOccurs="0" maxOccurs="1" type="ama:IT01" dfdl:initiator="1" dfdl:occursCountKind="implicit" />
>             <xs:element name="record" type="ama:Record" minOccurs="1" dfdl:occursCountKind="implicit" />
>         </xs:sequence>
>     </xs:complexType>
>
>     <xs:complexType name="Record">
>         <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
>             <xs:element name="it02" type="ama:IT02" dfdl:initiator="2" dfdl:occursCountKind="implicit" />
>             <xs:element name="it03" minOccurs="0" type="ama:IT03" dfdl:initiator="3" dfdl:occursCountKind="implicit" />
>             <xs:element name="it05" minOccurs="0" type="ama:IT05" dfdl:initiator="5" dfdl:occursCountKind="implicit" />
>         </xs:sequence>
>     </xs:complexType>
>
> </xs:schema>
>
>
> A file would look like this:
>
> 1......
> 2......
> 5......
>
> Elements it01 (line starting with '1') and it02 (line starting with '2')
> are parsed successfully. Element it03 is missing, but since it is optional
> it is OK. The problem is that the Daffodil raises an error saying it has
> not found initiator '3'. How to make the initiator optional?
>
> Patrick.
>

Re: Optional element with initiator

Posted by "Beckerle, Mike" <mb...@tresys.com>.
I see a couple issues here.

First your record element is not optional but you have dfdl:occursCountKind="implicit" on it, suggesting you wanted minOccurs="0" on it, or maxOccurs="unbounded",  but you didn't put that.

Same for it02 element, which probably wants minOccurs="0".

Second, your records type has a sequence with postfix newline separator.
Your inner sequence inside the record type *also* has postfix newline separator.

This means you will have TWO newlines required at the end of your data.

If you don't have two newlines, then it's going to backtrack and, while I can't construct a specific reason why it ends up giving you an error on it03, I can imagine that could happen.

I suggest removing the postfix from the inner sequence so it is only infix separated, and the final newline will come from the outer sequence.

Then I suggest making it02 element optional and record element optional or removing the dfdl:occursCountKind="implicit" from either or both of them.

Then try again. Post your results here again.

You may want to turn on the trace feature. You get TONS of sometimes obscure output from trace (fixing that is on our roadmap), but often one can wade through it and spot where things go wrong.

If your format has many many of these different record sub-types you will eventually want to switch away from using initiators to a choice using the dfdl:choiceDispatchKey property.

But we should get what you started with working properly first.




________________________________
From: Patrick Grandjean <p....@gmail.com>
Sent: Monday, February 3, 2020 2:23 PM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Optional element with initiator

Hi!

I'm trying to create my first DFDL schema to parse a fixed column format. The format may change from one line to the other depending on the first character of the line. Some lines are optional, therefore I use minOccurs="0" and dlfl:occursCountKind="implicit". Here is the schema:


<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
           xmlns:ex="http://example.com"
           xmlns:ama="http://www.amadeus.com"
           targetNamespace="http://www.amadeus.com"
           elementFormDefault="unqualified">

    <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
    <xs:include schemaLocation="IT01.dfdl.xsd" />
    <xs:include schemaLocation="IT02.dfdl.xsd" />
    <xs:include schemaLocation="IT03.dfdl.xsd" />
    <xs:include schemaLocation="IT05.dfdl.xsd" />

    <xs:annotation>
        <xs:appinfo source="http://www.ogf.org/dfdl/">
            <dfdl:format alignment="1"
                         alignmentUnits="bytes"
                         encoding="ASCII"
                         encodingErrorPolicy="replace"
                         escapeSchemeRef=""
                         floating="no"
                         ignoreCase="no"
                         initiatedContent="no"
                         initiator=""
                         leadingSkip="0"
                         lengthKind="implicit"
                         lengthUnits="bytes"
                         outputNewLine="%LF;"
                         representation="text"
                         separator=""
                         separatorSuppressionPolicy="never"
                         sequenceKind="ordered"
                         terminator=""
                         textBidi="no"
                         textPadKind="none"
                         textStringJustification="center"
                         textStringPadCharacter="%WSP;"
                         textTrimKind="padChar"
                         trailingSkip="0"
                         truncateSpecifiedLengthString="no"/>
        </xs:appinfo>
    </xs:annotation>

    <xs:element name="records" type="ama:Records" />

    <xs:complexType name="Records">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it01" minOccurs="0" maxOccurs="1" type="ama:IT01" dfdl:initiator="1" dfdl:occursCountKind="implicit" />
            <xs:element name="record" type="ama:Record" minOccurs="1" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="Record">
        <xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
            <xs:element name="it02" type="ama:IT02" dfdl:initiator="2" dfdl:occursCountKind="implicit" />
            <xs:element name="it03" minOccurs="0" type="ama:IT03" dfdl:initiator="3" dfdl:occursCountKind="implicit" />
            <xs:element name="it05" minOccurs="0" type="ama:IT05" dfdl:initiator="5" dfdl:occursCountKind="implicit" />
        </xs:sequence>
    </xs:complexType>

</xs:schema>

A file would look like this:

1......
2......
5......

Elements it01 (line starting with '1') and it02 (line starting with '2') are parsed successfully. Element it03 is missing, but since it is optional it is OK. The problem is that the Daffodil raises an error saying it has not found initiator '3'. How to make the initiator optional?

Patrick.