You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@daffodil.apache.org by Peter Kostouros <Pe...@awta.com.au> on 2019/12/04 05:35:29 UTC
Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Hi
I hope someone can point me in the right direction to help me understand behaviour I have seen with a particular schema when parsing a file.
I have a schema modelled on the NACHA schema files found in the DFDLSchemas/NACHA directory on github. In my case, with respect to optional (embedded) looping elements:
1. Parsing is unsuccessful when maxOccurs attribute is set to "unbounded" (Parse Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion failed: Not Control Record);
2. Parsing is successful when maxOccurs is limited to say "999".
Below is a snippet from the schema referred to above that results in error:
<!-- F LOOP -->
<xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element ref="F01_Record" minOccurs="0" />
<xs:element dfdl:lengthKind="implicit" name="F02_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element ref="F02_Record" minOccurs="0" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
I have attached schema files that demonstrate this issue, so I hope someone can advise me on what I should be correctly doing.
I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and both show similar behaviour; I have also seen this behaviour when running the files though the daffodil command line, with the following tunables
"unqualifiedPathStepPolicy" = "defaultNamespace"
"suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
Peter
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
RE: Daffodil parsing fails on optional elements when maxOccurs set
to "unbounded", passes when set to "999"
Posted by Peter Kostouros <Pe...@awta.com.au>.
Thanks!
Peter
From: Sloane, Brandon [mailto:bsloane@tresys.com]
Sent: Thursday, 5 December 2019 11:59 AM
To: Peter Kostouros <Pe...@awta.com.au>; users@daffodil.apache.org
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
The issue is that you have maxOccurs="unbounded" on elements which are potentially 0 bits long.
In particular, F_Record and B_Record. Both of those elements have only optional children. This means that they will never fail to parse. Instead they will succeed in parsing, but consume 0 bits. Because they can occur an unbounded number of times, Daffodil considers this to be an error, and backtracks (and subsequently throws an unrelated error down the line).
When maxOccurs is finite, then Daffodil will parse the 0 bits a finite number of times before resuming the parse normally.
The simplest solution to this, is to add an explicit assertion that F_Record and B_Record are non-empty:
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert>
{ dfdl:contentLength(.,'bits') gt 0 }
</dfdl:assert>
</xs:appinfo>
</xs:annotation>
Attached, you will find a version of pug_records.xsd that takes this approach.
While this is not technically a bug in Daffodil, it really should issue a warning when this situation arises. I have opened a ticket to that effect: https://issues.apache.org/jira/browse/DAFFODIL-2247
Given the above, you may be wondering why you do not see thousands of empty instances of F_Record when maxOccurs="9999". I believe this is the correct behavior in this case as defined by section 9.4.2.3, but I would need to read the spec very closely to be sure that this is not a bug in Daffodil.
Regards,
Brandon
________________________________
From: Peter Kostouros <Pe...@awta.com.au>>
Sent: Wednesday, December 4, 2019 4:52 PM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: RE: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I have attached a dataset that shows the problem (PUG.IN) as well as its corresponding parsed output when the schema has set maxOccurs limits on selected optional elements (PUG_999.IN.XML).
The F_LOOP records in file PUG.IN start with "31" and "32A".
Peter
From: Sloane, Brandon [mailto:bsloane@tresys.com]
Sent: Thursday, 5 December 2019 2:44 AM
To: users@daffodil.apache.org<ma...@daffodil.apache.org>
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
The only thing that stands out to me is that the error you are seeing should be coming from ControlRecord, which isn't part of the quoted schema. Other then that, I am not sure what the issue could be (unless your data actually parses more then 999 instances when unbounded is used).
Do you have example data that you can share which demonstrates the problem?
________________________________
From: Peter Kostouros <Pe...@awta.com.au>>
Sent: Wednesday, December 4, 2019 12:35 AM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I hope someone can point me in the right direction to help me understand behaviour I have seen with a particular schema when parsing a file.
I have a schema modelled on the NACHA schema files found in the DFDLSchemas/NACHA directory on github. In my case, with respect to optional (embedded) looping elements:
1. Parsing is unsuccessful when maxOccurs attribute is set to "unbounded" (Parse Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion failed: Not Control Record);
2. Parsing is successful when maxOccurs is limited to say "999".
Below is a snippet from the schema referred to above that results in error:
<!-- F LOOP -->
<xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element ref="F01_Record" minOccurs="0" />
<xs:element dfdl:lengthKind="implicit" name="F02_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element ref="F02_Record" minOccurs="0" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
I have attached schema files that demonstrate this issue, so I hope someone can advise me on what I should be correctly doing.
I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and both show similar behaviour; I have also seen this behaviour when running the files though the daffodil command line, with the following tunables
"unqualifiedPathStepPolicy" = "defaultNamespace"
"suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
Peter
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
Re: Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Posted by "Beckerle, Mike" <mb...@tresys.com>.
I looked it up in the DFDL Spec.
There is a bug in daffodil.
Section 16.6 says the parsing of the array must "terminate" when no-forward-progress is detected.
This is true regardless of whether the maxOccurs is unbounded or not.
So the fact that we're failing the surrounding element, not just terminating the array, is a bug.
________________________________
From: Beckerle, Mike <mb...@tresys.com>
Sent: Thursday, December 5, 2019 11:09 AM
To: Steve Lawrence <sl...@apache.org>; users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
If Daffodil speculatively parses a F-Record, attempting and failing to parse F01 and F02 sub-records, but those are optional, it doesn't fail here. It produces an "empty" F-record. Since this F-record is an array element of a variable-length array, it is "optional" and so nothing is placed into the infoset.
That ends the speculation about the element being present or not.
But then in the surrounding context, we acertain that we made zero forward progress. That's an infinite loop if the array is "unbounded". So when we detect no forward progress on the array, we explicitly fail with no forward progress.
But there is no point of uncertainty for "the whole array", just individual elements of the array, and we're outside of the point of uncertainty for an array element.
The next enclosing point of uncertainty is the one that the "no forward progress" hits.
This is a interesting point of interpretation of the DFDL Spec. The spec is clear that no-forward-progress is a parse error, but it is not clear as to which point of uncertainty this parse error affects. Or rather, I have to investigate further to see if the spec is clear as to whether the right behavior here is that no-forward-progress ends the array, or no forward progress behaves as it does now, which is outside of the scope of parsing an element of the array.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Thursday, December 5, 2019 7:46 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Stepping through the debugger, I'm seeing behavior that makes me wonder
if there's another part to this. The relevant data looks like this:
...
31...
32A...
32A...
32A...
32A...
79...
...
So there's a single F1_Record (31), four F2_Record's (32A), and a
C01_Record (79).
Daffodil successfully parses the F1 and F2 records, resulting in an
infoset like this:
<F_Records>
<F_Record>
<F01_Record>...</F01Record>
<F02_Records>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
</F02_Records>
</F_Record>
</F_Records>
It then tries to speculatively parse another F_Record since there can be
an unbounded number of these. Inside this F_Record it tries to parse an
F01_Record and an F02_Record, both of which fail since the next field is
79 (C01_Record), which is okay since they are optional.
Here's where I think things go wrong. Daffodil now needs to backtrack
and undo that second F_Record it incorrectly speculatively parsed. But
instead of undoing just the second F_Record it undoes the entire
F_Records element, including the fields that were successfully parsed,
and backtracks all the way back to the first 31 record. Instead of
undoing the entire F_Records, it should just undo the second F_Record.
Since it backtracked too far, it then continually tries to parse the
first 31 record, but that never works. And everything is optional so
Daffodil keeps chugging along, speculatively parsing and then undoing
the speculative parse.
Finally it gets to the Control_Record, which *is* required, and that
fails because 31 is not a Control_Record. And leads to the error about
Control_Record having an invalid valid.
So I think the underlying issue here is that when maxOccurs is unbounded
we get to a point where we parse 0 bytes, try to deal with no forward
progress back backtracking but end up backtrack too far.
Things work in the bounded case because it tries to speculatively parse
998 F_Records, those all fail (just like in the unbounded case), but
then it doesn't remove the entire F_Record.
On 12/4/19 7:59 PM, Sloane, Brandon wrote:
> The issue is that you have maxOccurs="unbounded" on elements which are
> potentially 0 bits long.
>
> In particular, F_Record and B_Record. Both of those elements have only optional
> children. This means that they will never fail to parse. Instead they will
> succeed in parsing, but consume 0 bits. Because they can occur an unbounded
> number of times, Daffodil considers this to be an error, and backtracks (and
> subsequently throws an unrelated error down the line).
>
> When maxOccurs is finite, then Daffodil will parse the 0 bits a finite number of
> times before resuming the parse normally.
>
> The simplest solution to this, is to add an explicit assertion that F_Record and
> B_Record are non-empty:
>
> <xs:annotation>
> <xs:appinfo source="http://www.ogf.org/dfdl/">
> <dfdl:assert>
> { dfdl:contentLength(.,'bits') gt 0 }
> </dfdl:assert>
> </xs:appinfo>
> </xs:annotation>
>
>
> Attached, you will find a version of pug_records.xsd that takes this approach.
>
> While this is not technically a bug in Daffodil, it really should issue a
> warning when this situation arises. I have opened a ticket to that effect:
> https://issues.apache.org/jira/browse/DAFFODIL-2247
>
> Given the above, you may be wondering why you do not see thousands of empty
> instances of F_Record when maxOccurs="9999". I believe this is the correct
> behavior in this case as defined by section 9.4.2.3, but I would need to read
> the spec very closely to be sure that this is not a bug in Daffodil.
>
> Regards,
> Brandon
>
> --------------------------------------------------------------------------------
> *From:* Peter Kostouros <Pe...@awta.com.au>
> *Sent:* Wednesday, December 4, 2019 4:52 PM
> *To:* users@daffodil.apache.org <us...@daffodil.apache.org>
> *Subject:* RE: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I have attached a dataset that shows the problem (PUG.IN) as well as its
> corresponding parsed output when the schema has set maxOccurs limits on selected
> optional elements (PUG_999.IN.XML).
>
> The F_LOOP records in file PUG.IN start with “31” and “32A”.
>
> **
>
> **
>
> Peter
>
> *From:*Sloane, Brandon [mailto:bsloane@tresys.com]
> *Sent:* Thursday, 5 December 2019 2:44 AM
> *To:* users@daffodil.apache.org
> *Subject:* Re: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> The only thing that stands out to me is that the error you are seeing should be
> coming from ControlRecord, which isn't part of the quoted schema. Other then
> that, I am not sure what the issue could be (unless your data actually parses
> more then 999 instances when unbounded is used).
>
> Do you have example data that you can share which demonstrates the problem?
>
> --------------------------------------------------------------------------------
>
> *From:*Peter Kostouros <Peter.Kostouros@awta.com.au
> <ma...@awta.com.au>>
> *Sent:* Wednesday, December 4, 2019 12:35 AM
> *To:* users@daffodil.apache.org <ma...@daffodil.apache.org>
> <users@daffodil.apache.org <ma...@daffodil.apache.org>>
> *Subject:* Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I hope someone can point me in the right direction to help me understand
> behaviour I have seen with a particular schema when parsing a file.
>
> I have a schema modelled on the NACHA schema files found in the
> DFDLSchemas/NACHA directory on github. In my case, with respect to optional
> (embedded) looping elements:
>
> 1.Parsing is unsuccessful when maxOccurs attribute is set to “unbounded” (Parse
> Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion
> failed: Not Control Record);
>
> 2.Parsing is successful when maxOccurs is limited to say “999”.
>
> Below is a snippet from the schema referred to above that results in error:
>
> <!-- F LOOP -->
>
> <xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0"
> maxOccurs="unbounded">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F01_Record" minOccurs="0" />
>
> <xs:element dfdl:lengthKind="implicit" name="F02_Records"
> minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F02_Record" minOccurs="0"
> maxOccurs="unbounded" />
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> I have attached schema files that demonstrate this issue, so I hope someone can
> advise me on what I should be correctly doing.
>
> I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and
> both show similar behaviour; I have also seen this behaviour when running the
> files though the daffodil command line, with the following tunables
>
> "unqualifiedPathStepPolicy" = "defaultNamespace"
>
> "suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
>
> Peter
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
Re: Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Posted by "Beckerle, Mike" <mb...@tresys.com>.
If Daffodil speculatively parses a F-Record, attempting and failing to parse F01 and F02 sub-records, but those are optional, it doesn't fail here. It produces an "empty" F-record. Since this F-record is an array element of a variable-length array, it is "optional" and so nothing is placed into the infoset.
That ends the speculation about the element being present or not.
But then in the surrounding context, we acertain that we made zero forward progress. That's an infinite loop if the array is "unbounded". So when we detect no forward progress on the array, we explicitly fail with no forward progress.
But there is no point of uncertainty for "the whole array", just individual elements of the array, and we're outside of the point of uncertainty for an array element.
The next enclosing point of uncertainty is the one that the "no forward progress" hits.
This is a interesting point of interpretation of the DFDL Spec. The spec is clear that no-forward-progress is a parse error, but it is not clear as to which point of uncertainty this parse error affects. Or rather, I have to investigate further to see if the spec is clear as to whether the right behavior here is that no-forward-progress ends the array, or no forward progress behaves as it does now, which is outside of the scope of parsing an element of the array.
________________________________
From: Steve Lawrence <sl...@apache.org>
Sent: Thursday, December 5, 2019 7:46 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Stepping through the debugger, I'm seeing behavior that makes me wonder
if there's another part to this. The relevant data looks like this:
...
31...
32A...
32A...
32A...
32A...
79...
...
So there's a single F1_Record (31), four F2_Record's (32A), and a
C01_Record (79).
Daffodil successfully parses the F1 and F2 records, resulting in an
infoset like this:
<F_Records>
<F_Record>
<F01_Record>...</F01Record>
<F02_Records>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
</F02_Records>
</F_Record>
</F_Records>
It then tries to speculatively parse another F_Record since there can be
an unbounded number of these. Inside this F_Record it tries to parse an
F01_Record and an F02_Record, both of which fail since the next field is
79 (C01_Record), which is okay since they are optional.
Here's where I think things go wrong. Daffodil now needs to backtrack
and undo that second F_Record it incorrectly speculatively parsed. But
instead of undoing just the second F_Record it undoes the entire
F_Records element, including the fields that were successfully parsed,
and backtracks all the way back to the first 31 record. Instead of
undoing the entire F_Records, it should just undo the second F_Record.
Since it backtracked too far, it then continually tries to parse the
first 31 record, but that never works. And everything is optional so
Daffodil keeps chugging along, speculatively parsing and then undoing
the speculative parse.
Finally it gets to the Control_Record, which *is* required, and that
fails because 31 is not a Control_Record. And leads to the error about
Control_Record having an invalid valid.
So I think the underlying issue here is that when maxOccurs is unbounded
we get to a point where we parse 0 bytes, try to deal with no forward
progress back backtracking but end up backtrack too far.
Things work in the bounded case because it tries to speculatively parse
998 F_Records, those all fail (just like in the unbounded case), but
then it doesn't remove the entire F_Record.
On 12/4/19 7:59 PM, Sloane, Brandon wrote:
> The issue is that you have maxOccurs="unbounded" on elements which are
> potentially 0 bits long.
>
> In particular, F_Record and B_Record. Both of those elements have only optional
> children. This means that they will never fail to parse. Instead they will
> succeed in parsing, but consume 0 bits. Because they can occur an unbounded
> number of times, Daffodil considers this to be an error, and backtracks (and
> subsequently throws an unrelated error down the line).
>
> When maxOccurs is finite, then Daffodil will parse the 0 bits a finite number of
> times before resuming the parse normally.
>
> The simplest solution to this, is to add an explicit assertion that F_Record and
> B_Record are non-empty:
>
> <xs:annotation>
> <xs:appinfo source="http://www.ogf.org/dfdl/">
> <dfdl:assert>
> { dfdl:contentLength(.,'bits') gt 0 }
> </dfdl:assert>
> </xs:appinfo>
> </xs:annotation>
>
>
> Attached, you will find a version of pug_records.xsd that takes this approach.
>
> While this is not technically a bug in Daffodil, it really should issue a
> warning when this situation arises. I have opened a ticket to that effect:
> https://issues.apache.org/jira/browse/DAFFODIL-2247
>
> Given the above, you may be wondering why you do not see thousands of empty
> instances of F_Record when maxOccurs="9999". I believe this is the correct
> behavior in this case as defined by section 9.4.2.3, but I would need to read
> the spec very closely to be sure that this is not a bug in Daffodil.
>
> Regards,
> Brandon
>
> --------------------------------------------------------------------------------
> *From:* Peter Kostouros <Pe...@awta.com.au>
> *Sent:* Wednesday, December 4, 2019 4:52 PM
> *To:* users@daffodil.apache.org <us...@daffodil.apache.org>
> *Subject:* RE: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I have attached a dataset that shows the problem (PUG.IN) as well as its
> corresponding parsed output when the schema has set maxOccurs limits on selected
> optional elements (PUG_999.IN.XML).
>
> The F_LOOP records in file PUG.IN start with “31” and “32A”.
>
> **
>
> **
>
> Peter
>
> *From:*Sloane, Brandon [mailto:bsloane@tresys.com]
> *Sent:* Thursday, 5 December 2019 2:44 AM
> *To:* users@daffodil.apache.org
> *Subject:* Re: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> The only thing that stands out to me is that the error you are seeing should be
> coming from ControlRecord, which isn't part of the quoted schema. Other then
> that, I am not sure what the issue could be (unless your data actually parses
> more then 999 instances when unbounded is used).
>
> Do you have example data that you can share which demonstrates the problem?
>
> --------------------------------------------------------------------------------
>
> *From:*Peter Kostouros <Peter.Kostouros@awta.com.au
> <ma...@awta.com.au>>
> *Sent:* Wednesday, December 4, 2019 12:35 AM
> *To:* users@daffodil.apache.org <ma...@daffodil.apache.org>
> <users@daffodil.apache.org <ma...@daffodil.apache.org>>
> *Subject:* Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I hope someone can point me in the right direction to help me understand
> behaviour I have seen with a particular schema when parsing a file.
>
> I have a schema modelled on the NACHA schema files found in the
> DFDLSchemas/NACHA directory on github. In my case, with respect to optional
> (embedded) looping elements:
>
> 1.Parsing is unsuccessful when maxOccurs attribute is set to “unbounded” (Parse
> Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion
> failed: Not Control Record);
>
> 2.Parsing is successful when maxOccurs is limited to say “999”.
>
> Below is a snippet from the schema referred to above that results in error:
>
> <!-- F LOOP -->
>
> <xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0"
> maxOccurs="unbounded">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F01_Record" minOccurs="0" />
>
> <xs:element dfdl:lengthKind="implicit" name="F02_Records"
> minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F02_Record" minOccurs="0"
> maxOccurs="unbounded" />
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> I have attached schema files that demonstrate this issue, so I hope someone can
> advise me on what I should be correctly doing.
>
> I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and
> both show similar behaviour; I have also seen this behaviour when running the
> files though the daffodil command line, with the following tunables
>
> "unqualifiedPathStepPolicy" = "defaultNamespace"
>
> "suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
>
> Peter
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
Re: Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Posted by Steve Lawrence <sl...@apache.org>.
Stepping through the debugger, I'm seeing behavior that makes me wonder
if there's another part to this. The relevant data looks like this:
...
31...
32A...
32A...
32A...
32A...
79...
...
So there's a single F1_Record (31), four F2_Record's (32A), and a
C01_Record (79).
Daffodil successfully parses the F1 and F2 records, resulting in an
infoset like this:
<F_Records>
<F_Record>
<F01_Record>...</F01Record>
<F02_Records>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
<F02_Record>...</F02_Record>
</F02_Records>
</F_Record>
</F_Records>
It then tries to speculatively parse another F_Record since there can be
an unbounded number of these. Inside this F_Record it tries to parse an
F01_Record and an F02_Record, both of which fail since the next field is
79 (C01_Record), which is okay since they are optional.
Here's where I think things go wrong. Daffodil now needs to backtrack
and undo that second F_Record it incorrectly speculatively parsed. But
instead of undoing just the second F_Record it undoes the entire
F_Records element, including the fields that were successfully parsed,
and backtracks all the way back to the first 31 record. Instead of
undoing the entire F_Records, it should just undo the second F_Record.
Since it backtracked too far, it then continually tries to parse the
first 31 record, but that never works. And everything is optional so
Daffodil keeps chugging along, speculatively parsing and then undoing
the speculative parse.
Finally it gets to the Control_Record, which *is* required, and that
fails because 31 is not a Control_Record. And leads to the error about
Control_Record having an invalid valid.
So I think the underlying issue here is that when maxOccurs is unbounded
we get to a point where we parse 0 bytes, try to deal with no forward
progress back backtracking but end up backtrack too far.
Things work in the bounded case because it tries to speculatively parse
998 F_Records, those all fail (just like in the unbounded case), but
then it doesn't remove the entire F_Record.
On 12/4/19 7:59 PM, Sloane, Brandon wrote:
> The issue is that you have maxOccurs="unbounded" on elements which are
> potentially 0 bits long.
>
> In particular, F_Record and B_Record. Both of those elements have only optional
> children. This means that they will never fail to parse. Instead they will
> succeed in parsing, but consume 0 bits. Because they can occur an unbounded
> number of times, Daffodil considers this to be an error, and backtracks (and
> subsequently throws an unrelated error down the line).
>
> When maxOccurs is finite, then Daffodil will parse the 0 bits a finite number of
> times before resuming the parse normally.
>
> The simplest solution to this, is to add an explicit assertion that F_Record and
> B_Record are non-empty:
>
> <xs:annotation>
> <xs:appinfo source="http://www.ogf.org/dfdl/">
> <dfdl:assert>
> { dfdl:contentLength(.,'bits') gt 0 }
> </dfdl:assert>
> </xs:appinfo>
> </xs:annotation>
>
>
> Attached, you will find a version of pug_records.xsd that takes this approach.
>
> While this is not technically a bug in Daffodil, it really should issue a
> warning when this situation arises. I have opened a ticket to that effect:
> https://issues.apache.org/jira/browse/DAFFODIL-2247
>
> Given the above, you may be wondering why you do not see thousands of empty
> instances of F_Record when maxOccurs="9999". I believe this is the correct
> behavior in this case as defined by section 9.4.2.3, but I would need to read
> the spec very closely to be sure that this is not a bug in Daffodil.
>
> Regards,
> Brandon
>
> --------------------------------------------------------------------------------
> *From:* Peter Kostouros <Pe...@awta.com.au>
> *Sent:* Wednesday, December 4, 2019 4:52 PM
> *To:* users@daffodil.apache.org <us...@daffodil.apache.org>
> *Subject:* RE: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I have attached a dataset that shows the problem (PUG.IN) as well as its
> corresponding parsed output when the schema has set maxOccurs limits on selected
> optional elements (PUG_999.IN.XML).
>
> The F_LOOP records in file PUG.IN start with “31” and “32A”.
>
> **
>
> **
>
> Peter
>
> *From:*Sloane, Brandon [mailto:bsloane@tresys.com]
> *Sent:* Thursday, 5 December 2019 2:44 AM
> *To:* users@daffodil.apache.org
> *Subject:* Re: Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> The only thing that stands out to me is that the error you are seeing should be
> coming from ControlRecord, which isn't part of the quoted schema. Other then
> that, I am not sure what the issue could be (unless your data actually parses
> more then 999 instances when unbounded is used).
>
> Do you have example data that you can share which demonstrates the problem?
>
> --------------------------------------------------------------------------------
>
> *From:*Peter Kostouros <Peter.Kostouros@awta.com.au
> <ma...@awta.com.au>>
> *Sent:* Wednesday, December 4, 2019 12:35 AM
> *To:* users@daffodil.apache.org <ma...@daffodil.apache.org>
> <users@daffodil.apache.org <ma...@daffodil.apache.org>>
> *Subject:* Daffodil parsing fails on optional elements when maxOccurs set to
> "unbounded", passes when set to "999"
>
> Hi
>
> I hope someone can point me in the right direction to help me understand
> behaviour I have seen with a particular schema when parsing a file.
>
> I have a schema modelled on the NACHA schema files found in the
> DFDLSchemas/NACHA directory on github. In my case, with respect to optional
> (embedded) looping elements:
>
> 1.Parsing is unsuccessful when maxOccurs attribute is set to “unbounded” (Parse
> Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion
> failed: Not Control Record);
>
> 2.Parsing is successful when maxOccurs is limited to say “999”.
>
> Below is a snippet from the schema referred to above that results in error:
>
> <!-- F LOOP -->
>
> <xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0"
> maxOccurs="unbounded">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F01_Record" minOccurs="0" />
>
> <xs:element dfdl:lengthKind="implicit" name="F02_Records"
> minOccurs="0">
>
> <xs:complexType>
>
> <xs:sequence>
>
> <xs:element ref="F02_Record" minOccurs="0"
> maxOccurs="unbounded" />
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> </xs:sequence>
>
> </xs:complexType>
>
> </xs:element>
>
> I have attached schema files that demonstrate this issue, so I hope someone can
> advise me on what I should be correctly doing.
>
> I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and
> both show similar behaviour; I have also seen this behaviour when running the
> files though the daffodil command line, with the following tunables
>
> "unqualifiedPathStepPolicy" = "defaultNamespace"
>
> "suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
>
> Peter
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
> This e-mail and any attachment is intended for the party to which it is
> addressed and may contain confidential information or be subject to professional
> privilege. Its transmission in not intended to place the contents into the
> public domain. If you have received this e-mail in error, please notify us
> immediately and delete the email and all copies. AWTA Ltd does not warrant that
> this e-mail is virus or error free. By opening this e-mail and any attachment
> the user assumes all responsibility for any loss or damage resulting from such
> action, whether or not caused by the negligence of AWTA Ltd. The contents of
> this e-mail and any attachments are subject to copyright and may not be
> reproduced, adapted or transmitted without the prior written permission of the
> copyright owner.
>
Re: Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Posted by "Sloane, Brandon" <bs...@tresys.com>.
The issue is that you have maxOccurs="unbounded" on elements which are potentially 0 bits long.
In particular, F_Record and B_Record. Both of those elements have only optional children. This means that they will never fail to parse. Instead they will succeed in parsing, but consume 0 bits. Because they can occur an unbounded number of times, Daffodil considers this to be an error, and backtracks (and subsequently throws an unrelated error down the line).
When maxOccurs is finite, then Daffodil will parse the 0 bits a finite number of times before resuming the parse normally.
The simplest solution to this, is to add an explicit assertion that F_Record and B_Record are non-empty:
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert>
{ dfdl:contentLength(.,'bits') gt 0 }
</dfdl:assert>
</xs:appinfo>
</xs:annotation>
Attached, you will find a version of pug_records.xsd that takes this approach.
While this is not technically a bug in Daffodil, it really should issue a warning when this situation arises. I have opened a ticket to that effect: https://issues.apache.org/jira/browse/DAFFODIL-2247
Given the above, you may be wondering why you do not see thousands of empty instances of F_Record when maxOccurs="9999". I believe this is the correct behavior in this case as defined by section 9.4.2.3, but I would need to read the spec very closely to be sure that this is not a bug in Daffodil.
Regards,
Brandon
________________________________
From: Peter Kostouros <Pe...@awta.com.au>
Sent: Wednesday, December 4, 2019 4:52 PM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: RE: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I have attached a dataset that shows the problem (PUG.IN) as well as its corresponding parsed output when the schema has set maxOccurs limits on selected optional elements (PUG_999.IN.XML).
The F_LOOP records in file PUG.IN start with “31” and “32A”.
Peter
From: Sloane, Brandon [mailto:bsloane@tresys.com]
Sent: Thursday, 5 December 2019 2:44 AM
To: users@daffodil.apache.org
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
The only thing that stands out to me is that the error you are seeing should be coming from ControlRecord, which isn't part of the quoted schema. Other then that, I am not sure what the issue could be (unless your data actually parses more then 999 instances when unbounded is used).
Do you have example data that you can share which demonstrates the problem?
________________________________
From: Peter Kostouros <Pe...@awta.com.au>>
Sent: Wednesday, December 4, 2019 12:35 AM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I hope someone can point me in the right direction to help me understand behaviour I have seen with a particular schema when parsing a file.
I have a schema modelled on the NACHA schema files found in the DFDLSchemas/NACHA directory on github. In my case, with respect to optional (embedded) looping elements:
1. Parsing is unsuccessful when maxOccurs attribute is set to “unbounded” (Parse Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion failed: Not Control Record);
2. Parsing is successful when maxOccurs is limited to say “999”.
Below is a snippet from the schema referred to above that results in error:
<!-- F LOOP -->
<xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element ref="F01_Record" minOccurs="0" />
<xs:element dfdl:lengthKind="implicit" name="F02_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element ref="F02_Record" minOccurs="0" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
I have attached schema files that demonstrate this issue, so I hope someone can advise me on what I should be correctly doing.
I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and both show similar behaviour; I have also seen this behaviour when running the files though the daffodil command line, with the following tunables
"unqualifiedPathStepPolicy" = "defaultNamespace"
"suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
Peter
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
RE: Daffodil parsing fails on optional elements when maxOccurs set
to "unbounded", passes when set to "999"
Posted by Peter Kostouros <Pe...@awta.com.au>.
Hi
I have attached a dataset that shows the problem (PUG.IN) as well as its corresponding parsed output when the schema has set maxOccurs limits on selected optional elements (PUG_999.IN.XML).
The F_LOOP records in file PUG.IN start with "31" and "32A".
Peter
From: Sloane, Brandon [mailto:bsloane@tresys.com]
Sent: Thursday, 5 December 2019 2:44 AM
To: users@daffodil.apache.org
Subject: Re: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
The only thing that stands out to me is that the error you are seeing should be coming from ControlRecord, which isn't part of the quoted schema. Other then that, I am not sure what the issue could be (unless your data actually parses more then 999 instances when unbounded is used).
Do you have example data that you can share which demonstrates the problem?
________________________________
From: Peter Kostouros <Pe...@awta.com.au>>
Sent: Wednesday, December 4, 2019 12:35 AM
To: users@daffodil.apache.org<ma...@daffodil.apache.org> <us...@daffodil.apache.org>>
Subject: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I hope someone can point me in the right direction to help me understand behaviour I have seen with a particular schema when parsing a file.
I have a schema modelled on the NACHA schema files found in the DFDLSchemas/NACHA directory on github. In my case, with respect to optional (embedded) looping elements:
1. Parsing is unsuccessful when maxOccurs attribute is set to "unbounded" (Parse Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion failed: Not Control Record);
2. Parsing is successful when maxOccurs is limited to say "999".
Below is a snippet from the schema referred to above that results in error:
<!-- F LOOP -->
<xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element ref="F01_Record" minOccurs="0" />
<xs:element dfdl:lengthKind="implicit" name="F02_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element ref="F02_Record" minOccurs="0" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
I have attached schema files that demonstrate this issue, so I hope someone can advise me on what I should be correctly doing.
I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and both show similar behaviour; I have also seen this behaviour when running the files though the daffodil command line, with the following tunables
"unqualifiedPathStepPolicy" = "defaultNamespace"
"suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
Peter
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
Re: Daffodil parsing fails on optional elements when maxOccurs set to
"unbounded", passes when set to "999"
Posted by "Sloane, Brandon" <bs...@tresys.com>.
The only thing that stands out to me is that the error you are seeing should be coming from ControlRecord, which isn't part of the quoted schema. Other then that, I am not sure what the issue could be (unless your data actually parses more then 999 instances when unbounded is used).
Do you have example data that you can share which demonstrates the problem?
________________________________
From: Peter Kostouros <Pe...@awta.com.au>
Sent: Wednesday, December 4, 2019 12:35 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Daffodil parsing fails on optional elements when maxOccurs set to "unbounded", passes when set to "999"
Hi
I hope someone can point me in the right direction to help me understand behaviour I have seen with a particular schema when parsing a file.
I have a schema modelled on the NACHA schema files found in the DFDLSchemas/NACHA directory on github. In my case, with respect to optional (embedded) looping elements:
1. Parsing is unsuccessful when maxOccurs attribute is set to “unbounded” (Parse Error: Failed to populate ControlRecord[1]. Cause: Parse Error: Assertion failed: Not Control Record);
2. Parsing is successful when maxOccurs is limited to say “999”.
Below is a snippet from the schema referred to above that results in error:
<!-- F LOOP -->
<xs:element dfdl:lengthKind="implicit" name="F_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element dfdl:lengthKind="implicit" name="F_Record" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element ref="F01_Record" minOccurs="0" />
<xs:element dfdl:lengthKind="implicit" name="F02_Records" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element ref="F02_Record" minOccurs="0" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
I have attached schema files that demonstrate this issue, so I hope someone can advise me on what I should be correctly doing.
I am using the Daffodil 2.4.0 release as well as the 2.5 snapshot JAVA APIs, and both show similar behaviour; I have also seen this behaviour when running the files though the daffodil command line, with the following tunables
"unqualifiedPathStepPolicy" = "defaultNamespace"
"suppressSchemaDefinitionWarnings" = "multipleChoiceBranches noEmptyDefault"
Peter
This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.